Compare commits

..

28 Commits

Author SHA1 Message Date
overtrue f4266a4576 refactor(protocols): use owned object reads and deletion
Route both GetObject calls, HeadObject and DeleteObject to the owned use cases. Keep the request facts and millisecond timestamp behavior. PutObject stays unchanged; 12 operations and 13 trait calls remain.

The client edge is an interface in the layer guard. Its public re-export is covered too, and app-to-client imports are rejected by fixtures.

Runtime wiring: +177/-101 non-test lines. The new matrix has 18 negative and 13 positive operation cases, plus the extended identity-copy control.

Validation: 28 protocol tests passed in both default and gateway-stack modes; both all-target Clippy modes passed. Script tests, layer controls and focused architecture/footprint/test guards passed. Five source faults failed at the intended claims, range, Head ETag, deletion-witness and version assertions. The precision oracles failed before the millisecond projection was added.

Runtime and Clippy ran on e9da1d309; initial controls and faults ran on 2308dee4b2. Rebase to 00ecde125 changed none of the seven task files or gateway pins. CI and HTTP transport were not checked.
2026-11-16 12:11:38 +08:00
overtrue 00ecde1250 feat(gateway): dispatch console routes through native Axum
Use the existing console router when the gateway console is enabled. Keep lambda and extension precedence, the legacy selection, and outer middleware assembly unchanged. The common body conversion stays at the edge.

Add test-only assets and one contract with 10 native and 52 excluded routing probes. Non-test source: +89/-16. The restored run passed 22 console/stack tests on e9da1d309; both Clippy modes, script-tests and prepush-quick passed.

native_console_route_contract went red for three independent mutations: remove the native branch (service-call counts), widen the shared prefix (neighboring S3 path), and route every claim to console (anonymous admin refusal). Restoring the sources triggered a real product recompile.

Full outer HTTP wire checks and a production UI bundle were not tested. Admin, STS and website migration remain separate work.
2026-11-16 11:47:40 +08:00
overtrue e9da1d3096 feat(gateway): serve DeleteObjects through the owned engine
Share the batch deletion engine between legacy and native edges with owned inputs and results. Keep exact per-entry authorization, version and retention handling, ordered errors, quiet responses, metrics, and audit aggregation. Preserve frozen resource resolution and rejected-key echoes.

On 2308dee4 with gateway 44f9a242, the focused suite passed 208 tests before and after deliberate mutations. Both rustfs Clippy configurations, script tests, test wiring, formatting, and prepush guards passed. Four added runtime tests cover mixed successes and per-entry refusals, frozen fields, and rejected-key state.

Five targeted tests failed when batch target clearing, denial error mapping, and raw-key echo were changed. Sources were restored byte for byte and product artifacts recompiled.

The legacy edge matched 62 observed baseline responses including all headers, XML and object states. Native decoder and expected-owner differences are tracked separately. Actual webhook records matched for mixed and quiet requests across the baseline and both candidate stacks.

Full workspace tests and CI were not run. Verification is scoped to this migration increment.
2026-11-16 10:24:21 +08:00
overtrue 2308dee4b2 fix(gateway): preserve object retention date offsets
Use the validated request offset and full timestamp precision when projecting PutObject and PutObjectRetention dates into authorization and storage.

Signed requests verify exact stored dates and version isolation for positive and negative offsets and UTC. Policy tests cover denied dates and the typed date supplied by PutObject authorization.

On b2b4f0c4, 175 focused tests and six raw xl.meta checks passed. The UTC-only control and five targeted mutations failed at their intended assertions. Both Clippy feature configurations passed.
2026-11-16 08:39:36 +08:00
overtrue b2b4f0c491 feat(gateway): serve SelectObjectContent natively
Share GetObject authorization and the existing Select query producer across both stacks. Carry live events through the typed response with the projected SSE headers. The producer retains its single deadline, snapshot checks, bounded channel and receiver cancellation.

On c5b4a8577, the restored build passed 207 focused tests, both RustFS Clippy feature modes, formatting, script tests and test wiring. Signed socket probes matched all 17 legacy cases and 12 gateway cases. Five gateway refusals omit the legacy default message, tracked in rustfs/backlog#2684.

The fault build passed all 98 controls. Bypassing authorization failed on both stacks. Removing receiver-close handling, dropping the terminal Records permit, skipping the Stats fence, replacing an error with End, and dropping progress or scan range each failed the corresponding test.

Verification used local stores and SSE-C. The full workspace suite and external KMS were outside the local run.
2026-11-16 07:29:41 +08:00
overtrue c5b4a85778 feat(gateway): serve RestoreObject through the owned use case
Share the authorization snapshot and restore admission between both stacks.
Preserve HTTP 200 for both success paths and emit ObjectRestorePost on new
admission only.

Verified on c0763a43 with 114 focused tests, four serialized ILM tests, and
16 signed wire responses per stack. Faults covered shared authorization,
generation capture and admission, SELECT rejection, worker parameters and
liveness, and both audit and event paths. Cluster lock loss and mixed-version
rollout remain unverified.
2026-11-16 06:30:48 +08:00
overtrue 4ae180ea40 feat(gateway): serve object legal hold and retention writes
Share each operation's authorization and metadata write with the legacy edge.
Keep their request ordering, version and bucket-incarnation fences, replication
completion, and notification suppression. Retention checks the locked metadata
again before committing a change.

The two stacks were exercised through signed HTTP requests and independent
disk metadata reads. Faults in authorization, body projection, version
selection, counting, auditing, notification suppression, scanner invalidation,
the bucket fence, and the commit-time retention check reached their assertions.
2026-11-16 06:19:43 +08:00
overtrue c0763a43f5 feat(gateway): serve bucket object lock configuration
Share the bucket Object Lock write between the legacy and gateway edges. Keep lock configuration and versioning as sequential writes against the authorized bucket incarnation, followed by peer reload, site replication and scanner invalidation.

Cover eight refusal or failure cases and three successful cases with durable metadata and signed requests. The two stacks run the same request cases and refusal matrix.

The mutation run detected both missing write fences, swallowed peer failure, propagated site failure, early or absent scanner invalidation, missing audit, wrong authorization action or resource, dropped request body and incorrect operation counting.

Validation used debug builds on d461908bf. The final lint pass elides an equivalent lifetime in a test helper. Peer and site failures were injected locally, with real multi-site delivery left unverified.
2026-11-16 05:35:18 +08:00
overtrue d461908bf0 feat(gateway): serve bucket replication configuration writes
Route PutBucketReplication and DeleteBucketReplication through the owned bucket use case. Share site checks, metadata fencing, target cleanup recovery, and audit completion with the legacy edge.

Signed requests exercise 28 refusals and six accepted edits across both stacks. The direct storage tests cover six failure paths and two successful edits. Twenty-four faults turned their expected disk, authorization, locking, callback, audit, or metric assertions red.

The restored gateway-stack build passed 132 selected tests on 75b293f09, including both HTTP stacks. Both Clippy variants and the mutation cohort passed on fb3e2818c. The fixed legacy CLI matched all six response records. The standalone default runtime suite, multi-site delivery, and CI were unverified.
2026-11-16 04:32:24 +08:00
overtrue 5b8876ad0e feat(gateway): serve bucket policy, encryption, and tagging writes
Share the owned write paths across both S3 stacks. Preserve the policy
document bytes, default KMS key resolution, bucket tag validation, and
incarnation fences before reload and replication callbacks.

Add three successful writes and three rejected stale writes on disk.
Extend the two-stack refusal matrix with six successful and 24 denied
requests. Cover the encryption and tag validation order before lookup.

The fault cohort exercised the action and resource checks, value
validation and order, raw policy bytes, default KMS readback, legacy
policy count, durable writes, incarnation fences, reload, replication,
audit, and scanner invalidation. Each selected assertion failed.

Local checks used base 75b293f09. Fixed CLI comparisons covered legacy
responses, audit entries, and operation counts. Live peer replication,
external KMS providers, and CI were outside the local checks.
2026-11-16 04:24:32 +08:00
overtrue 75b293f094 feat(gateway): serve encryption, policy, and tagging deletes
Share the three generation-fenced deletes across both S3 stacks. Keep
their existing IAM actions and use the shared IAM scope for tag deletion.

Add six focused tests covering 21 refusals and nine successes, and add the
three operations to the refusal matrix. Substituting IAM actions and
buckets, removing the tagging scope, and suppressing the gateway counter
fail the authorization and metrics checks. The storage tests detect a
lost fence, a skipped write, missing reload or replication calls, omitted
scanner invalidation, and a changed audit status.

Verified on 4294c41cc with scoped nextest, both Clippy modes, script tests,
quick checks, and local CLI comparisons. Peer network delivery was
outside the local checks.
2026-11-16 03:21:06 +08:00
overtrue fb3e2818c7 feat(gateway): serve bucket ACL and logging writes
Route PutBucketAcl and PutBucketLogging through shared bucket use cases.
Preserve the ACL response for header-only requests and the bucket lookup
before the XML policy refusal. Write logging configuration using the
authorized bucket incarnation and the existing persistence format.

Verify 111 focused tests on 4294c41cc, including the storage access suite,
and run both Clippy modes, script tests, and local wire comparisons. The
wire matrix exercises 15 refusal cases and seven successful writes on
each stack.

Change both IAM actions and resources, audit statuses, and counters in
the fault cohort. Reverse ACL lookup and XML policy checks, drop policy
presence, corrupt logging bytes, reverse logging conversion order, and
remove the incarnation guard. The regressions detect each fault, and
the stale-authorization test reads the replacement bucket from disk.
2026-11-16 03:08:10 +08:00
overtrue 4294c41ccf feat(gateway): serve bucket public access block operations
Route Get, Put, and DeletePublicAccessBlock through shared bucket use
cases. Preserve the persisted switch values, mutation incarnation fence,
reload behavior, audit status, and missing-configuration response.

Add six focused tests covering 25 refusal cases and 17 success cases
across both stacks and storage. Extend the refusal and counter matrix
for all three operations.

Substitute each IAM action and bucket, corrupt each operation's value,
change the missing-config status, and suppress each edge counter. The
round-trip and matrix tests reject those faults. Put and Delete tests
reject a missing incarnation fence, the wrong reload mode, and wrong
audit status. Read the durable replacement-bucket bytes after disabling
both fences.

Verify on 92050cdca with scoped nextest, both Clippy modes, script tests,
and local wire tests. Peer reload delivery was outside these local tests.
2026-11-16 02:10:24 +08:00
overtrue 3cb0aa137e feat(gateway): serve core bucket configuration mutations
Serve accelerate, request-payment and website writes, including website
deletion, through shared use cases. Both S3 edges enforce the bucket
policy and the metadata incarnation fence before changing stored data.

Verified on 92050cdca with formatting, Clippy on both RustFS feature
sets, script tests, 21 RustFS tests and 27 operation and metrics tests.
The fixed legacy CLI supplied 42 HTTP comparison cases per stack.

The wire assertions caught altered IAM actions and resources, missing
denial audits and operation counters, and skipped website validation.
The stale-incarnation test caught changes to the replacement bucket's
disk bytes. The lookup-order test caught early configuration conversion.

Refs rustfs/backlog#2734
2026-11-16 02:07:32 +08:00
Hauser Zhi a8cf62f8f3 fix(metrics): label grouped fdatasync batches accurately (#9446)
* fix(metrics): label grouped fdatasync batches accurately

Record the serialized user-space fdatasync coordinator as grouped and reserve parallel for independent per-file fanout. This keeps performance dashboards from conflating two different dispatch models.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): separate grouping from timeout race

Keep the same-disk batching test focused on queue aggregation. Claimed-follower timeout ownership remains covered by its dedicated deterministic regression, avoiding a valid pre-claim timeout race in the batching fixture.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): accept borrowed multipart guard calls

Count both owned String and borrowed &str spellings of the same reserved-prefix guard. UploadPart now obtains borrowed bucket and key values from its typed target, while the production guard remains in place.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 12:51:57 +00:00
Hauser Zhi d0fd6425e4 feat(rpc): guard stale generations before dispatch (#9435)
* perf(grpc): attribute rebuilds and bound bulk responses

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): attribute rebuild success after publication

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): add exact HTTP/2 ping timeout diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): run refusal matrix on large stack

Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.

Keep the original test name and all refusal-order assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): pin nixl-sys workspace source

Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.

Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): authenticate msgpack peer capability

Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(grpc): bind peer capability to connection generations (#9433)

* feat(grpc): track physical connection generations

Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): bind msgpack capability to connections

Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): guard stale generations before dispatch

Reject generation-bound requests before inner transport call or first payload frame when tonic readiness observes a different physical connection, while leaving ordinary mutations unguarded until the send-policy follow-up.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): add per-peer msgpack rollout gate

Add the default-off peer-capability environment gate, cached evaluation, fixed-cardinality send-decision metrics, and rollout constraints for stable peer-affine connections.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): apply per-peer msgpack send policy

Capture one capability decision per eligible RPC, bind peer-mode requests to boot and connection generations, and rebuild at most one dual-write request only after a locally proven pre-dispatch rejection.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): verify msgpack peer canary evidence

Add a strict three-phase four-node evidence verifier with raw Prometheus and workload receipts, adversarial tests, script-test wiring, and the operational manifest contract.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): align batch read manifest encoder

Match the BatchReadVersion compatibility manifest to the request-owned encoder expression used by the production send path.

Restore the owning ECStore source guard while preserving wire and runtime behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(rpc): orchestrate msgpack peer canary

Run the fixed mixed/all-new/rollback four-node matrix with prebuilt images, real workload and readback, raw Prometheus capture, strict manifest generation, and adversarial fake integration tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(rpc): verify canary image and telemetry identities

* test(rpc): bind canary evidence provenance

Scope canary metrics to one injected cluster identity and bind candidate, baseline, and per-phase running containers to immutable image IDs and exact build revisions.

Reject foreign-cluster metrics, stale revision labels, same-image tags, and phase image mismatches.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(table-catalog): accept borrowed multipart guard calls

Count the exact borrowed bucket/key guard call introduced by UploadPart target ownership while preserving the four-production-call contract.

Cover the borrowed spelling in both the positive fixture and the test-only false-positive regression.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
2026-11-15 12:37:25 +00:00
overtrue 92050cdca6 feat(gateway): serve PutObjectTagging through the shared use case
Share XML tag authorization and storage execution across both stacks.
Preserve the requested version, replication-target proxy, and metadata
completion order.

The dual-stack tests exercise 37 refusal and 13 success scenarios.
The restored integration run passed 148 tests on 26db9b19a, and the
proxy cleanup verification passed 134 run, 134 passed.
Both feature configurations passed Clippy, and script tests passed.

Fault injection caught native and shared authorization bypasses,
missing XML facts, and incorrect action, resource and tag key facts.
It also caught duplicate counting, incorrect persisted tags, missing
or duplicate success events, and event or dirty publication on a miss.

Live replication targets and ECStore write faults remain unverified.
2026-11-16 00:29:49 +08:00
overtrue 26db9b19af feat(gateway): serve core bucket configuration reads
Serve bucket acceleration, request payment, website, and logging
configuration through shared persisted read paths. Keep each operation's
existing authorization and legacy response projection. Append distinct
metric labels without shifting existing indexes.

The scoped RustFS, S3 operation, and metrics suite passed 283 tests on
993464fbb. The integrated configuration and refusal suites passed 21
tests on 292974baf. Both feature modes passed all-targets Clippy.

Standalone checks compared 23 successful and 17 refused legacy requests
against cfdd4b268 and exercised the gateway answers on the same data.
Action, resource, anonymous, projection, default, missing-bucket, counter,
and audit faults failed their original assertions. Metric registry tests
also detected duplicate labels, changed index order, and an added event.
Verification is scoped to these local suites and wire probes.
2026-11-15 23:29:07 +08:00
overtrue 292974baff feat(gateway): serve bucket CORS, encryption and tagging reads
Share bucket-scoped authorization and persisted configuration reads across
both stacks. Carry the stored configurations in gateway responses and
preserve the operation-specific errors for missing configs.

On RustFS 993464fbb, 120 scoped tests passed, including both refusal
matrices and six new tests with 24 refusal and 12 success cases.
Live HTTP checks covered response documents, operation counts and
authorization-denial audit records. Both feature configurations passed
Clippy, and formatting, script tests and source guards passed.

Changing each IAM action or resource admitted a forbidden read.
Duplicating each edge count and changing each stored output failed its
test. Returning 404 for missing encryption failed the 400 assertion.
Suppressing gateway denial audit removed all nine expected records.
Runtime verification used single-node debug servers.
2026-11-15 22:48:03 +08:00
overtrue 993464fbbb feat(gateway): serve GetObjectAttributes through the owned path
Share the read use case and both IAM decisions across the legacy and
gateway handlers. Keep attribute selection, stored metadata and
pagination in the use case, and encode the result at each edge.

The 6f93ac5f control matched 55 legacy wire cases and the restored
sources passed 139 scoped tests. Both stacks exercised 59 signed
requests, including 31 legacy refusals and 32 gateway refusals.

Seventeen injected faults each turned a targeted assertion red. They
cover the four IAM actions, resource and version binding, empty-version
selection, handler authorization, SSE-C, selection, both checksum paths,
paging, ETag, Last-Modified, counting and notification suppression.
2026-11-15 21:41:16 +08:00
overtrue cfdd4b2686 feat(gateway): serve PutBucketNotificationConfiguration
Share one owned write path across the gateway and legacy edges. Project
runtime destinations at each edge and defer consumption of the persisted
conversion result until after filter validation. Keep IAM scope, the
incarnation fence, and the existing validate, write, reload, clear, and
add order. Count the request at its edge and audit it in the use case.

Add three refusal tests and four tests that exercise success and failure. The legacy HTTP baseline passed on 97a7639f. On 6f93ac5f the
38-test control passed. Six fault injections exposed wrong IAM action,
wrong IAM resource, a missing incarnation fence, duplicate counting,
the wrong audit status, and missing reload callbacks. The first three
changed the stored bytes guarded by their refusal tests.

The restored tree rebuilt and passed all 38 scoped tests. Both package
Clippy configurations, script tests, test wiring, and prepush passed.
Cross-node reload delivery and remote audit sink delivery were outside
the local test scope.
2026-11-15 21:18:37 +08:00
overtrue 6f93ac5f47 feat(gateway): serve AbortMultipartUpload
Use the same authorization and audited use case on both S3 stacks. Bind
the gateway claim to the owner selected under the upload write lock
before staged parts can be removed. Preserve the existing generation,
lifecycle and quorum checks.

Verified the fixed legacy baseline, the integrated native wire flow, 212
focused tests and both Clippy modes. Fault checks exercised owner
selection, deletion and orphan handling, scoped IAM, generation
transport, audit, operation counts and dirty usage.

Audit delivery and separate fault injection of the existing final
lock-loss and metadata-incarnation fences were outside this verification.
2026-11-15 18:05:23 +08:00
overtrue 620fdc286a feat(gateway): serve CopyObject through the owned path
Resolve the source from the gateway read proof. Share source and
destination authorization, tag checks, and the existing storage copy
between both S3 stacks.

Checked the legacy wire baseline, both CopyObject wire contracts, and the
access and refusal suites. Nine temporary faults triggered the intended
assertions. The restored build passed seven focused tests with gateway
75757292, and Clippy passed with and without gateway-stack.
2026-11-15 18:02:37 +08:00
overtrue 58310c569b chore: pin RustFS to gateway mainflow update
Refresh the four RustFS gateway dependency pins to gateway commit 75757292b9617080cbb1bb7f880db8da9b34912c. Update the gateway facade and types versions; keep stream and HTTP versions unchanged.

Verification: cargo check --locked -p rustfs --all-targets --features gateway-stack, layer dependency guard, architecture migration guard, and prepush-quick passed. No Rust sources or tests changed.
2026-11-15 17:42:10 +08:00
overtrue e36c7452ac test: cover DeleteBucketLifecycle in the refusal matrix
Seed a lifecycle configuration so the admitted request removes real state.

Verified the matrix census and both refusal matrices in the CopyObject development checkout on 93947f22c.
2026-11-15 16:41:34 +08:00
overtrue 48e97253b5 feat(gateway): serve ListParts through the owned path
Share authorization and pagination across both S3 edges. Resolve the
native upload claim using the owner from the same quorum-selected
metadata as the parts page. Preserve typed storage errors and write
the page through each edge's response projection.

The scoped suite includes all 98 storage access tests, real upload
sessions, stored-owner combinations, both wire paths, and the storage
API. Twenty-three fault modes exercised the owner, authorization,
lookup, error, counter, and projection assertions.

Validated 168 focused tests after restoring and recompiling the candidate
on c2b5fbb plus the isolated fixture cleanup. The owner matrix covers
three positive and six negative cases; each wire IAM matrix covers one
positive and seven negative cases. Both feature configurations passed
Clippy. Full migration acceptance lanes remain pending.
2026-11-15 16:24:31 +08:00
overtrue ce5bd8b117 test: isolate CreateMultipartUpload in the refusal matrix
Abort the session returned by the successful matrix call through the
fixture store so later pagination assertions see their own sessions.

Both listing boundary tests failed on c2b5fbb and passed with the
fixture cleanup. Both refusal matrices passed with the same cleanup.
2026-11-15 15:32:04 +08:00
overtrue 93947f22ce feat(gateway): serve DeleteBucketLifecycle through the owned use case
Share the incarnation-fenced lifecycle deletion across both edges, preserving reload and site-hook ordering and one audit record.

Validated 37 focused tests on 77da63397, both all-target Clippy configurations, script tests and wiring guards. Thirteen effective faults covered deletion, authorization wiring, incarnation fences, counting, reload and hook ordering, and audit behavior. Live peer delivery and final migration acceptance lanes remain unverified.
2026-11-15 15:28:43 +08:00
173 changed files with 18885 additions and 6602 deletions
+2
View File
@@ -39,6 +39,8 @@ script-tests: ## Run shell script tests
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_grpc_window.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_msgpack_peer_canary.py
./scripts/test_internode_msgpack_peer_canary_runner.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
Generated
+23 -23
View File
@@ -255,7 +255,7 @@ version = "1.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "40c48f72fd53cd289104fc64099abca73db4166ad86ea0b4341abe65af83dadc"
dependencies = [
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -266,7 +266,7 @@ checksum = "291e6a250ff86cd4a820112fb8898808a366d8f9f58ce16d1f538353ad55747d"
dependencies = [
"anstyle",
"once_cell_polyfill",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -2015,7 +2015,7 @@ dependencies = [
"maybe-owned",
"rustix",
"rustix-linux-procfs",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
"winx",
]
@@ -4447,7 +4447,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e35d6e78f4cebb90e51d543db8fe41528851d6deeef6ef21ebd4bfdc1e784776"
dependencies = [
"libc",
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
@@ -5844,7 +5844,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "20fd6de4ccfcc187e38bc21cfa543cb5a302cb86a8b114eb7f0bf0dc9f8ac00f"
dependencies = [
"io-lifetimes 3.0.1",
"windows-sys 0.60.2",
"windows-sys 0.59.0",
]
[[package]]
@@ -9026,7 +9026,7 @@ dependencies = [
"once_cell",
"socket2",
"tracing",
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
@@ -10266,8 +10266,8 @@ dependencies = [
[[package]]
name = "rustfs-gateway"
version = "0.63.0"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
version = "0.66.0"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"arc-swap",
"bytes",
@@ -10313,8 +10313,8 @@ dependencies = [
[[package]]
name = "rustfs-gateway-core"
version = "0.60.0"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
version = "0.63.0"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"bytes",
"futures-core",
@@ -10335,7 +10335,7 @@ dependencies = [
[[package]]
name = "rustfs-gateway-http"
version = "0.1.4"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"hmac 0.13.0",
"http 1.5.0",
@@ -10353,7 +10353,7 @@ dependencies = [
[[package]]
name = "rustfs-gateway-macros"
version = "0.1.4"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"proc-macro2",
"quote",
@@ -10363,7 +10363,7 @@ dependencies = [
[[package]]
name = "rustfs-gateway-sig"
version = "0.15.0"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"hmac 0.13.0",
"http 1.5.0",
@@ -10378,7 +10378,7 @@ dependencies = [
[[package]]
name = "rustfs-gateway-stream"
version = "0.2.0"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"bitflags 2.13.2",
"bytes",
@@ -10388,8 +10388,8 @@ dependencies = [
[[package]]
name = "rustfs-gateway-types"
version = "0.40.0+aws.2026-10-02"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
version = "0.42.0+aws.2026-10-02"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"bytes",
"crc-fast",
@@ -10412,7 +10412,7 @@ dependencies = [
[[package]]
name = "rustfs-gateway-xml"
version = "0.4.0"
source = "git+https://github.com/rustfs/gateway.git?rev=2a6b8d1f7663db978d8784ab3ea331cade19237b#2a6b8d1f7663db978d8784ab3ea331cade19237b"
source = "git+https://github.com/rustfs/gateway.git?rev=44f9a2422d745ffba952347ec91b0332ef2667a4#44f9a2422d745ffba952347ec91b0332ef2667a4"
dependencies = [
"quick-xml 0.41.0",
]
@@ -11642,7 +11642,7 @@ dependencies = [
"errno",
"libc",
"linux-raw-sys",
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
@@ -11724,7 +11724,7 @@ dependencies = [
"security-framework",
"security-framework-sys",
"webpki-root-certs",
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
@@ -12541,7 +12541,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c3d1e2c7f27f8d4cb10542a02c49005dbd6e93095799d6f3be745fae9f8fedd4"
dependencies = [
"libc",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -12692,7 +12692,7 @@ dependencies = [
"cfg-if",
"libc",
"psm",
"windows-sys 0.60.2",
"windows-sys 0.61.2",
]
[[package]]
@@ -13003,7 +13003,7 @@ dependencies = [
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
@@ -14118,7 +14118,7 @@ version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys 0.52.0",
"windows-sys 0.59.0",
]
[[package]]
+4 -4
View File
@@ -151,14 +151,14 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
# the s3s-facing edge of the rustfs binary.
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.40.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.2.0", default-features = false }
rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "44f9a2422d745ffba952347ec91b0332ef2667a4", version = "0.42.0", default-features = false }
rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "44f9a2422d745ffba952347ec91b0332ef2667a4", version = "0.2.0", default-features = false }
# The gateway framework itself, consumed for its extension traits by the ring-2
# adapter. `default-features = false` keeps the `server` feature, and with it the
# server crate, out of the build.
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", version = "0.63.0", default-features = false }
rustfs-gateway = { git = "https://github.com/rustfs/gateway.git", rev = "44f9a2422d745ffba952347ec91b0332ef2667a4", version = "0.66.0", default-features = false }
# Test-only: builds the `EffectiveHost` a host-resolution test queries with.
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "2a6b8d1f7663db978d8784ab3ea331cade19237b", default-features = false }
rustfs-gateway-http = { git = "https://github.com/rustfs/gateway.git", rev = "44f9a2422d745ffba952347ec91b0332ef2667a4", default-features = false }
# Async Runtime and Networking
async-channel = "2.5.0"
+9
View File
@@ -121,9 +121,17 @@ pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
/// Permit current per-peer capability proof to replace fleet-wide confirmation.
///
/// This remains ineffective unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY`] is also enabled. Unknown,
/// unsupported, revoked, or stale connection generations keep dual-writing JSON.
pub const ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: bool = false;
// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY);
/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
@@ -525,6 +533,7 @@ mod tests {
ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
"RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
);
assert_eq!(ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY, "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY");
}
#[test]
@@ -171,70 +171,6 @@ impl DataUsageBucketObservation {
}
impl DataUsageInfo {
/// Hide values that cannot be bound to the current storage-owned identity.
///
/// `incarnations` contains canonical non-nil identities read from storage;
/// a missing entry means identity was unavailable. Neither legacy values
/// without identity nor metadata read failures establish confirmed zero.
/// Changed responses are display-only and cannot authorize quota or writes.
/// If retained counters overflow, all values remain unknown.
/// Returns the number of canonical values removed.
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
let original_count = self.buckets_usage.len();
let original_observation_count = self.usage_snapshot_bucket_observations.len();
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
incarnations.get(bucket).is_some_and(|current| {
!current.is_empty()
&& current != "00000000-0000-0000-0000-000000000000"
&& usage.bucket_incarnation.as_ref() == Some(current)
})
};
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
matches_current(bucket, &observation.usage)
&& observation
.sources
.iter()
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
});
if original_count == self.buckets_usage.len()
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
{
return 0;
}
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
self.buckets_usage.clear();
self.usage_snapshot_bucket_observations.clear();
[0; 4]
});
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_admin_projection = true;
original_count.saturating_sub(self.buckets_usage.len())
}
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
@@ -321,110 +321,3 @@ fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
#[test]
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
let mut candidate = snapshot();
let original = candidate.clone();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
assert_eq!(candidate, original);
assert!(candidate.is_complete_bucket_usage_snapshot());
}
#[test]
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
for identity in [
None,
Some(""),
Some("corrupt"),
Some("00000000-0000-0000-0000-000000000000"),
Some("22222222-2222-4222-8222-222222222222"),
] {
let mut candidate = snapshot();
candidate.usage_snapshot_bucket_observations.clear();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = identity.map(str::to_string);
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
1,
"identity {identity:?}"
);
assert!(!candidate.buckets_usage.contains_key("bucket"));
assert!(!candidate.bucket_sizes.contains_key("bucket"));
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
assert!(!candidate.is_valid_partial_snapshot());
}
for current in ["", "00000000-0000-0000-0000-000000000000"] {
let mut candidate = snapshot();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some(current.to_string());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
1
);
assert!(candidate.usage_snapshot_admin_projection);
}
let mut candidate = snapshot();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(7));
candidate.buckets_count = 2;
candidate.bucket_sizes.insert("cold".to_string(), 7);
candidate.calculate_totals();
candidate.replication_info.insert("target".to_string(), Default::default());
candidate.tier_stats = Some(Default::default());
candidate.unknown_tier_stats = Some(Default::default());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
1
);
assert_eq!(candidate.buckets_usage.len(), 1);
assert_eq!(candidate.buckets_usage["cold"].size, 7);
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
assert_eq!(
(
candidate.buckets_count,
candidate.objects_total_size,
candidate.objects_total_count,
candidate.versions_total_count
),
(1, 7, 7, 7)
);
assert!(candidate.replication_info.is_empty());
assert!(candidate.tier_stats.is_none());
assert!(candidate.unknown_tier_stats.is_none());
assert!(candidate.usage_snapshot_admin_projection);
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_overflow_leaves_totals_unknown() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
candidate.buckets_usage.insert("small".to_string(), usage(1));
candidate.buckets_count = 3;
let current = HashMap::from([
("large".to_string(), INCARNATION.to_string()),
("small".to_string(), INCARNATION.to_string()),
]);
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&current), 3);
assert!(candidate.buckets_usage.is_empty());
assert!(candidate.bucket_sizes.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
}
+2 -2
View File
@@ -453,8 +453,8 @@ pub mod data_usage {
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_compression_total_from_memory,
load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
+586 -7
View File
@@ -55,7 +55,7 @@ use std::{
task::{Context, Poll},
time::Duration,
};
use tonic::service::interceptor::InterceptedService;
use tonic::{codegen::StdError, service::interceptor::InterceptedService};
use tower::Service;
use tracing::debug;
use uuid::Uuid;
@@ -84,7 +84,7 @@ pub(crate) async fn scanner_control_time_out_client(
let connection_state = channel.connection_state();
let interceptor = interceptor.with_connection_state(connection_state.clone());
let channel = ReplayScopeChannel::new(
BatchLockDeadlineChannel::new(channel),
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -104,7 +104,7 @@ pub async fn heal_control_time_out_client(
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(
BatchLockDeadlineChannel::new(channel),
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -123,7 +123,7 @@ pub async fn tier_mutation_control_time_out_client(
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(
BatchLockDeadlineChannel::new(channel),
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -150,7 +150,7 @@ pub async fn node_service_time_out_client_for_class(
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(
BatchLockDeadlineChannel::new(channel),
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -159,6 +159,39 @@ pub async fn node_service_time_out_client_for_class(
.max_encoding_message_size(max_message_size))
}
/// Build an eligible request client and capture its peer capability exactly once for this RPC.
pub(crate) async fn node_service_time_out_client_for_class_with_msgpack_policy(
addr: &str,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<
(
NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>,
MsgpackSendPolicy,
),
Box<dyn Error>,
> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let audience = interceptor.replay_scope_audience();
let channel = get_internode_channel_for_class(addr, class).await?;
let connection_state = channel.connection_state();
let policy =
prepare_msgpack_send_policy(audience.as_deref(), connection_state.as_ref().and_then(GrpcConnectionState::snapshot));
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
audience,
connection_state,
);
Ok((
NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size),
policy,
))
}
pub(crate) async fn node_service_delete_recovery_client(
addr: &str,
interceptor: TonicInterceptor,
@@ -171,7 +204,7 @@ pub(crate) async fn node_service_delete_recovery_client(
let connection_state = channel.connection_state();
let interceptor = interceptor.with_connection_state(connection_state.clone());
let channel = ReplayScopeChannel::new(
BatchLockDeadlineChannel::new(channel),
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -291,7 +324,238 @@ pub struct ReplayScopeChannel<S> {
}
/// The channel type used by internode clients after v2 authentication and replay-scope handling.
pub type AuthenticatedChannel = ReplayScopeChannel<BatchLockDeadlineChannel<InternodeChannel>>;
pub type AuthenticatedChannel = ReplayScopeChannel<ConnectionGenerationChannel<BatchLockDeadlineChannel<InternodeChannel>>>;
const CONNECTION_GENERATION_PENDING: u8 = 0;
const CONNECTION_GENERATION_DISPATCHED: u8 = 1;
const CONNECTION_GENERATION_REJECTED: u8 = 2;
#[derive(Debug)]
struct ConnectionGenerationGuardInner {
expected: GrpcConnectionSnapshot,
expected_boot_epoch: Option<Uuid>,
outcome: AtomicU8,
}
/// A request-owned proof that its encoded body targets one physical connection generation.
#[derive(Clone, Debug)]
pub(crate) struct ConnectionGenerationGuard(Arc<ConnectionGenerationGuardInner>);
impl ConnectionGenerationGuard {
pub(crate) fn new(expected: GrpcConnectionSnapshot) -> Self {
Self(Arc::new(ConnectionGenerationGuardInner {
expected,
expected_boot_epoch: None,
outcome: AtomicU8::new(CONNECTION_GENERATION_PENDING),
}))
}
fn for_peer_capability(expected: GrpcConnectionSnapshot, boot_epoch: Uuid) -> Self {
Self(Arc::new(ConnectionGenerationGuardInner {
expected,
expected_boot_epoch: Some(boot_epoch),
outcome: AtomicU8::new(CONNECTION_GENERATION_PENDING),
}))
}
fn expected_peer_capability(&self) -> Option<(GrpcConnectionSnapshot, Uuid)> {
self.0.expected_boot_epoch.map(|boot_epoch| (self.0.expected, boot_epoch))
}
fn reject_before_dispatch(&self) {
let _ = self.0.outcome.compare_exchange(
CONNECTION_GENERATION_PENDING,
CONNECTION_GENERATION_REJECTED,
Ordering::AcqRel,
Ordering::Acquire,
);
}
fn validate_before_call(&self, current: Option<GrpcConnectionSnapshot>) -> Result<(), StaleConnectionGenerationError> {
if self.0.outcome.load(Ordering::Acquire) == CONNECTION_GENERATION_PENDING && current == Some(self.0.expected) {
return Ok(());
}
self.reject_before_dispatch();
Err(StaleConnectionGenerationError::new(self.0.expected, current))
}
fn begin_payload(&self, current: Option<GrpcConnectionSnapshot>) -> Result<(), tonic::Status> {
if current != Some(self.0.expected) {
self.reject_before_dispatch();
return Err(stale_connection_generation_status());
}
self.0
.outcome
.compare_exchange(
CONNECTION_GENERATION_PENDING,
CONNECTION_GENERATION_DISPATCHED,
Ordering::AcqRel,
Ordering::Acquire,
)
.map(|_| ())
.map_err(|_| stale_connection_generation_status())
}
fn rejected_before_dispatch(&self) -> bool {
self.0.outcome.load(Ordering::Acquire) == CONNECTION_GENERATION_REJECTED
}
}
fn stale_connection_generation_status() -> tonic::Status {
tonic::Status::aborted("internode connection generation changed before request dispatch")
}
#[derive(Debug)]
struct StaleConnectionGenerationError {
expected: GrpcConnectionSnapshot,
current: Option<GrpcConnectionSnapshot>,
status: tonic::Status,
}
impl StaleConnectionGenerationError {
fn new(expected: GrpcConnectionSnapshot, current: Option<GrpcConnectionSnapshot>) -> Self {
Self {
expected,
current,
status: stale_connection_generation_status(),
}
}
}
impl std::fmt::Display for StaleConnectionGenerationError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
formatter,
"internode connection changed before dispatch (expected {:?}, current {:?})",
self.expected, self.current
)
}
}
impl Error for StaleConnectionGenerationError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
Some(&self.status)
}
}
pin_project_lite::pin_project! {
#[project = ConnectionGenerationFutureProj]
#[doc(hidden)]
pub enum ConnectionGenerationFuture<F, Response> {
Inner { #[pin] future: F },
Rejected { #[pin] future: std::future::Ready<Result<Response, StdError>> },
}
}
impl<F, Response, InnerError> Future for ConnectionGenerationFuture<F, Response>
where
F: Future<Output = Result<Response, InnerError>>,
InnerError: Into<StdError>,
{
type Output = Result<Response, StdError>;
fn poll(self: Pin<&mut Self>, context: &mut Context<'_>) -> Poll<Self::Output> {
match self.project() {
ConnectionGenerationFutureProj::Inner { future } => future.poll(context).map_err(Into::into),
ConnectionGenerationFutureProj::Rejected { future } => future.poll(context),
}
}
}
/// Rejects a stale encoded request before dispatching any body data to tonic's channel.
#[derive(Clone, Debug)]
pub struct ConnectionGenerationChannel<S> {
inner: S,
connection_state: Option<GrpcConnectionState>,
}
impl<S> ConnectionGenerationChannel<S> {
fn new(inner: S, connection_state: Option<GrpcConnectionState>) -> Self {
Self { inner, connection_state }
}
}
impl<S> Service<HttpRequest<tonic::body::Body>> for ConnectionGenerationChannel<S>
where
S: Service<HttpRequest<tonic::body::Body>>,
S::Error: Into<StdError>,
{
type Response = S::Response;
type Error = StdError;
type Future = ConnectionGenerationFuture<S::Future, S::Response>;
fn poll_ready(&mut self, context: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
self.inner.poll_ready(context).map_err(Into::into)
}
fn call(&mut self, mut request: HttpRequest<tonic::body::Body>) -> Self::Future {
let Some(guard) = request.extensions().get::<ConnectionGenerationGuard>().cloned() else {
return ConnectionGenerationFuture::Inner {
future: self.inner.call(request),
};
};
let current = self.connection_state.as_ref().and_then(GrpcConnectionState::snapshot);
if let Err(error) = guard.validate_before_call(current) {
return ConnectionGenerationFuture::Rejected {
future: std::future::ready(Err(Box::new(error))),
};
}
let body = std::mem::replace(request.body_mut(), tonic::body::Body::empty());
*request.body_mut() = tonic::body::Body::new(ConnectionGenerationBody {
inner: body,
guard,
connection_state: self.connection_state.clone(),
checked: false,
failed: false,
});
ConnectionGenerationFuture::Inner {
future: self.inner.call(request),
}
}
}
struct ConnectionGenerationBody {
inner: tonic::body::Body,
guard: ConnectionGenerationGuard,
connection_state: Option<GrpcConnectionState>,
checked: bool,
failed: bool,
}
impl http_body::Body for ConnectionGenerationBody {
type Data = bytes::Bytes;
type Error = tonic::Status;
fn poll_frame(
mut self: Pin<&mut Self>,
context: &mut Context<'_>,
) -> Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
if self.failed {
return Poll::Ready(None);
}
let frame = std::task::ready!(Pin::new(&mut self.inner).poll_frame(context));
if let Some(Ok(frame)) = &frame
&& frame.is_data()
&& !self.checked
{
let current = self.connection_state.as_ref().and_then(GrpcConnectionState::snapshot);
if let Err(error) = self.guard.begin_payload(current) {
self.failed = true;
return Poll::Ready(Some(Err(error)));
}
self.checked = true;
}
Poll::Ready(frame)
}
fn is_end_stream(&self) -> bool {
self.failed || self.inner.is_end_stream()
}
fn size_hint(&self) -> http_body::SizeHint {
self.inner.size_hint()
}
}
/// Gate lock-batch payloads after the transport's internal readiness queues.
/// Other RPC bodies pass through without an additional wrapper.
@@ -515,6 +779,149 @@ fn should_request_msgpack_capability(state: PeerReplayStateSnapshot) -> bool {
}
}
fn validate_peer_capability_guard(guard: &ConnectionGenerationGuard, state: PeerReplayStateSnapshot) {
let Some((expected_connection, expected_boot_epoch)) = guard.expected_peer_capability() else {
return;
};
let capability_is_current = state.connection == Some(expected_connection)
&& state.replay.boot_epoch == Some(expected_boot_epoch)
&& matches!(
state.msgpack_capability,
Some(PeerMsgpackCapability::Capable {
boot_epoch,
connection_generation,
}) if boot_epoch == expected_boot_epoch
&& connection_generation == expected_connection.generation
);
if !capability_is_current {
guard.reject_before_dispatch();
}
}
#[derive(Clone, Debug)]
enum MsgpackSendMode {
DualWrite,
ManualFleet,
PeerCapable { guard: ConnectionGenerationGuard },
}
/// One eligible RPC's immutable JSON/msgpack decision and optional pre-dispatch guard.
#[derive(Clone, Debug)]
pub(crate) struct MsgpackSendPolicy {
mode: MsgpackSendMode,
}
impl MsgpackSendPolicy {
#[cfg(test)]
pub(crate) fn for_test(omit_eligible_json: bool) -> Self {
Self {
mode: if omit_eligible_json {
MsgpackSendMode::ManualFleet
} else {
MsgpackSendMode::DualWrite
},
}
}
fn dual_write(reason: &'static str) -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics().record_msgpack_send_decision("dual_write", reason);
Self {
mode: MsgpackSendMode::DualWrite,
}
}
fn manual_fleet() -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics()
.record_msgpack_send_decision("msgpack_only", "manual_fleet");
Self {
mode: MsgpackSendMode::ManualFleet,
}
}
fn peer_capable(connection: GrpcConnectionSnapshot, boot_epoch: Uuid) -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics()
.record_msgpack_send_decision("msgpack_only", "peer_capable");
Self {
mode: MsgpackSendMode::PeerCapable {
guard: ConnectionGenerationGuard::for_peer_capability(connection, boot_epoch),
},
}
}
pub(crate) fn omit_eligible_json(&self) -> bool {
!matches!(self.mode, MsgpackSendMode::DualWrite)
}
pub(crate) fn guard_request<T>(&self, request: &mut tonic::Request<T>) {
if let MsgpackSendMode::PeerCapable { guard } = &self.mode {
request.extensions_mut().insert(guard.clone());
}
}
/// Switch one peer-capability attempt to dual-write only after a proven local rejection.
pub(crate) fn retry_dual_write_after_local_rejection(&mut self) -> bool {
let rejected = matches!(
&self.mode,
MsgpackSendMode::PeerCapable { guard } if guard.rejected_before_dispatch()
);
if rejected {
*self = Self::dual_write("peer_revoked");
}
rejected
}
}
fn prepare_msgpack_send_policy(audience: Option<&str>, connection: Option<GrpcConnectionSnapshot>) -> MsgpackSendPolicy {
prepare_msgpack_send_policy_with_gates(
audience,
connection,
rustfs_protos::internode_rpc_msgpack_only(),
rustfs_protos::internode_rpc_msgpack_peer_capability_enabled(),
)
}
fn prepare_msgpack_send_policy_with_gates(
audience: Option<&str>,
connection: Option<GrpcConnectionSnapshot>,
manual_fleet: bool,
peer_capability: bool,
) -> MsgpackSendPolicy {
if manual_fleet {
return MsgpackSendPolicy::manual_fleet();
}
if !peer_capability {
return MsgpackSendPolicy::dual_write("not_requested");
}
let (Some(audience), Some(connection)) = (audience, connection) else {
return MsgpackSendPolicy::dual_write("peer_unknown");
};
let Ok(states) = PEER_REPLAY_STATES.lock() else {
return MsgpackSendPolicy::dual_write("peer_unknown");
};
let snapshot = states.snapshot(audience, Some(connection));
let (capable_epoch, revoked) = match (snapshot.replay.boot_epoch, snapshot.msgpack_capability) {
(
Some(boot_epoch),
Some(PeerMsgpackCapability::Capable {
boot_epoch: capability_epoch,
connection_generation,
}),
) if boot_epoch == capability_epoch && connection.generation == connection_generation => (Some(boot_epoch), false),
(_, Some(PeerMsgpackCapability::Revoked { connection_generation })) if connection.generation == connection_generation => {
(None, true)
}
_ => (None, false),
};
drop(states);
if let Some(boot_epoch) = capable_epoch {
MsgpackSendPolicy::peer_capable(connection, boot_epoch)
} else if revoked {
MsgpackSendPolicy::dual_write("peer_revoked")
} else {
MsgpackSendPolicy::dual_write("peer_unknown")
}
}
pub(crate) fn clear_peer_replay_state_for_addr(addr: &str) -> std::io::Result<()> {
let uri = addr
.parse::<Uri>()
@@ -687,7 +1094,17 @@ where
.get::<PeerReplayStateSnapshot>()
.copied()
.unwrap_or_default();
if let Some(guard) = request.extensions().get::<ConnectionGenerationGuard>() {
validate_peer_capability_guard(guard, sent_state);
}
let msgpack_capability_requested = authenticated && should_request_msgpack_capability(sent_state);
if msgpack_capability_requested
&& request.uri().path() == "/node_service.NodeService/Ping"
&& request.extensions().get::<ConnectionGenerationGuard>().is_none()
&& let Some(expected) = sent_state.connection
{
request.extensions_mut().insert(ConnectionGenerationGuard::new(expected));
}
if let (Some(audience), Some(challenge)) = (self.audience.as_deref(), challenge) {
// The challenge is independently HMAC-authenticated by the response proof. It is not
// part of v2 so old peers ignore it, while a new peer can safely advertise its epoch.
@@ -1011,6 +1428,106 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[tokio::test]
async fn connection_generation_guard_rejects_only_local_predispatch_races() {
use http_body_util::BodyExt as _;
use std::sync::atomic::AtomicUsize;
#[derive(Clone)]
struct DrainBodyService {
calls: Arc<AtomicUsize>,
payloads: Arc<AtomicUsize>,
}
impl Service<HttpRequest<tonic::body::Body>> for DrainBodyService {
type Response = HttpResponse<tonic::body::Body>;
type Error = tonic::Status;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _context: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, request: HttpRequest<tonic::body::Body>) -> Self::Future {
self.calls.fetch_add(1, Ordering::SeqCst);
let payloads = Arc::clone(&self.payloads);
Box::pin(async move {
let mut body = request.into_body();
match body.frame().await {
Some(Ok(frame)) if frame.is_data() => {
payloads.fetch_add(1, Ordering::SeqCst);
Ok(HttpResponse::new(tonic::body::Body::empty()))
}
Some(Err(error)) => Err(error),
_ => Err(tonic::Status::internal("guard test body did not produce payload data")),
}
})
}
}
fn request(guard: ConnectionGenerationGuard) -> HttpRequest<tonic::body::Body> {
let mut request =
HttpRequest::new(tonic::body::Body::new(http_body_util::Full::new(bytes::Bytes::from_static(b"payload"))));
request.extensions_mut().insert(guard);
request
}
fn has_local_generation_source(status: &tonic::Status) -> bool {
let mut source = status.source();
while let Some(error) = source {
if error.downcast_ref::<StaleConnectionGenerationError>().is_some() {
return true;
}
source = error.source();
}
false
}
let state = GrpcConnectionState::new();
let first = state.record_connected();
let calls = Arc::new(AtomicUsize::new(0));
let payloads = Arc::new(AtomicUsize::new(0));
let service = DrainBodyService {
calls: Arc::clone(&calls),
payloads: Arc::clone(&payloads),
};
let mut channel = ConnectionGenerationChannel::new(service, Some(state.clone()));
let rejected_before_call = ConnectionGenerationGuard::new(first);
let second = state.record_connected();
let error = channel
.call(request(rejected_before_call.clone()))
.await
.expect_err("stale generation must fail before inner call");
let status = tonic::Status::from_error(error);
assert_eq!(status.code(), tonic::Code::Aborted);
assert!(has_local_generation_source(&status));
assert!(rejected_before_call.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 0);
assert_eq!(payloads.load(Ordering::SeqCst), 0);
assert!(!has_local_generation_source(&tonic::Status::aborted(status.message())));
let rejected_before_payload = ConnectionGenerationGuard::new(second);
let pending = channel.call(request(rejected_before_payload.clone()));
let third = state.record_connected();
let error = pending
.await
.expect_err("generation change before body poll must reject payload");
assert_eq!(tonic::Status::from_error(error).code(), tonic::Code::Aborted);
assert!(rejected_before_payload.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert_eq!(payloads.load(Ordering::SeqCst), 0);
let dispatched = ConnectionGenerationGuard::new(third);
channel
.call(request(dispatched.clone()))
.await
.expect("matching generation dispatches payload");
assert!(!dispatched.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 2);
assert_eq!(payloads.load(Ordering::SeqCst), 1);
}
#[tokio::test]
async fn delete_recovery_channel_uses_its_scoped_deadline() {
use bytes::Bytes;
@@ -2056,6 +2573,68 @@ mod tests {
clear_peer_capability(audience);
}
#[test]
fn msgpack_send_policy_requires_exact_capability_and_retries_only_local_rejection() {
let audience = "msgpack-send-policy-test:9000";
let boot_epoch = Uuid::new_v4();
let connection = GrpcConnectionSnapshot {
channel_id: 21,
generation: 201,
};
clear_peer_capability(audience);
let unknown = PEER_REPLAY_STATES
.lock()
.expect("peer replay registry")
.snapshot(audience, Some(connection));
apply_peer_capabilities_response_for_connection(
audience.to_string(),
unknown,
Ok(authenticated_peer_response(boot_epoch, true)),
Some(Ok(authenticated_msgpack_response(boot_epoch, true))),
Some(connection),
);
let mut peer_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(connection), false, true);
assert!(peer_policy.omit_eligible_json());
let mut request = tonic::Request::new(());
peer_policy.guard_request(&mut request);
assert!(request.extensions().get::<ConnectionGenerationGuard>().is_some());
let MsgpackSendMode::PeerCapable { guard } = &peer_policy.mode else {
panic!("proved peer must use guarded msgpack-only policy");
};
let guard = guard.clone();
{
let mut states = PEER_REPLAY_STATES.lock().expect("peer replay registry");
states.set_msgpack_capability(
audience,
Some(connection),
PeerMsgpackCapability::Revoked {
connection_generation: connection.generation,
},
);
}
let revoked = PEER_REPLAY_STATES
.lock()
.expect("peer replay registry")
.snapshot(audience, Some(connection));
validate_peer_capability_guard(&guard, revoked);
assert!(peer_policy.retry_dual_write_after_local_rejection());
assert!(!peer_policy.omit_eligible_json());
assert!(!peer_policy.retry_dual_write_after_local_rejection());
let replacement = GrpcConnectionSnapshot {
channel_id: connection.channel_id,
generation: connection.generation + 1,
};
let unknown_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(replacement), false, true);
assert!(!unknown_policy.omit_eligible_json());
let manual_policy = prepare_msgpack_send_policy_with_gates(None, None, true, false);
assert!(manual_policy.omit_eligible_json());
let disabled_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(connection), false, false);
assert!(!disabled_policy.omit_eligible_json());
clear_peer_capability(audience);
}
#[test]
fn clear_peer_replay_state_for_addr_removes_normalized_audience() {
let audience = "clear-peer-replay-state-test:9000";
+345 -223
View File
@@ -13,8 +13,9 @@
// limitations under the License.
use crate::cluster::rpc::client::{
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
AuthenticatedChannel, MsgpackSendPolicy, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
node_service_time_out_client_for_class_with_msgpack_policy,
};
use crate::cluster::rpc::http_auth::{current_rpc_background_request, with_rpc_background_request};
use crate::cluster::rpc::internode_data_transport::{
@@ -2453,6 +2454,19 @@ impl RemoteDisk {
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn get_client_with_msgpack_policy(&self) -> Result<(NodeRpcClient, MsgpackSendPolicy)> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
node_service_time_out_client_for_class_with_msgpack_policy(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
ChannelClass::Control,
)
.await
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
/// Client for large `bytes`-carrying RPCs, including native pool metadata recovery inventories.
/// Routes onto the isolated bulk channel pool so large transfers cannot head-of-line block
/// lock/health RPCs (grpc-optimization P1). Falls back to the control channel when isolation
@@ -2493,6 +2507,19 @@ impl RemoteDisk {
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn get_bulk_client_with_msgpack_policy(&self) -> Result<(NodeRpcClient, MsgpackSendPolicy)> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
node_service_time_out_client_for_class_with_msgpack_policy(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
ChannelClass::Bulk,
)
.await
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn disk_ref(&self) -> String {
(*self.id.lock().await)
.map(|id| id.to_string())
@@ -2535,8 +2562,8 @@ fn encode_file_info_versions_msgpack(value: &FileInfoVersions) -> Result<Vec<u8>
/// JSON compatibility string for a dual-encoded (`_bin` + text) request field. Returns an empty
/// string only when msgpack-only mode and its explicit fleet confirmation guard are both enabled;
/// otherwise the legacy JSON encoding is retained for old peers.
fn compat_json<T: Serialize>(value: &T) -> Result<String> {
if rustfs_protos::internode_rpc_msgpack_only() {
fn compat_json<T: Serialize>(value: &T, policy: &MsgpackSendPolicy) -> Result<String> {
if policy.omit_eligible_json() {
return Ok(String::new());
}
Ok(serde_json::to_string(value)?)
@@ -2810,26 +2837,12 @@ impl RemoteDisk {
|| async {
let prepare_started = rustfs_io_metrics::put_stage_timer();
let collect_prepare_timing = prepare_started.is_some();
let json_encode_started = collect_prepare_timing.then(Instant::now);
let file_info = compat_json(fi);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_ENCODE,
json_encode_started,
);
let file_info = file_info?;
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_BYTES,
file_info.len(),
);
}
let msgpack_encode_started = collect_prepare_timing.then(Instant::now);
let file_info_bin = encode_file_info_msgpack(fi);
let file_info_bin = Bytes::from(encode_file_info_msgpack(fi)?);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_MSGPACK_ENCODE,
msgpack_encode_started,
);
let file_info_bin = file_info_bin?;
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_MSGPACK_BYTES,
@@ -2837,35 +2850,57 @@ impl RemoteDisk {
);
}
let get_client_started = collect_prepare_timing.then(Instant::now);
let client = self.get_client().await;
let client = self.get_client_with_msgpack_policy().await;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_GET_CLIENT,
get_client_started,
);
let mut client = client?;
let mut request = Request::new(RenameDataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk: self.endpoint.to_string(),
src_volume: src_volume.to_string(),
src_path: src_path.to_string(),
file_info,
dst_volume: dst_volume.to_string(),
dst_path: dst_path.to_string(),
file_info_bin: file_info_bin.into(),
scanner_publication_lease_token: scanner_publication_lease_token
.map(|token| token.as_bytes().to_vec().into())
.unwrap_or_default(),
});
let canonical_body_digest_started = collect_prepare_timing.then(Instant::now);
let incarnation_bound = !request.get_ref().bucket_incarnation_id.is_empty();
let (mut client, mut policy) = client?;
let bucket_incarnation_id = crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
.map(|scope| Bytes::copy_from_slice(scope.incarnation.as_bytes()))
.unwrap_or_default();
let incarnation_bound = !bucket_incarnation_id.is_empty();
let quota_fenced = rustfs_utils::http::metadata_compat::contains_key_str(
&fi.metadata,
crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX,
);
let canonical_body_digest = (|| -> Result<()> {
let scanner_publication_lease = scanner_publication_lease_token
.map(|token| Bytes::copy_from_slice(token.as_bytes()))
.unwrap_or_default();
let rpc_started = rustfs_io_metrics::put_stage_timer();
let mut retried_dual_write = false;
let response = loop {
let attempt_prepare_started = if retried_dual_write {
rustfs_io_metrics::put_stage_timer()
} else {
prepare_started
};
let json_encode_started = collect_prepare_timing.then(Instant::now);
let file_info = compat_json(fi, &policy)?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_ENCODE,
json_encode_started,
);
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_BYTES,
file_info.len(),
);
}
let mut request = Request::new(RenameDataRequest {
bucket_incarnation_id: bucket_incarnation_id.clone(),
disk: self.endpoint.to_string(),
src_volume: src_volume.to_string(),
src_path: src_path.to_string(),
file_info,
dst_volume: dst_volume.to_string(),
dst_path: dst_path.to_string(),
file_info_bin: file_info_bin.clone(),
scanner_publication_lease_token: scanner_publication_lease.clone(),
});
let canonical_body_digest_started = collect_prepare_timing.then(Instant::now);
let canonical_body = rustfs_protos::canonical_rename_data_request_body(request.get_ref());
policy.guard_request(&mut request);
if pool_meta_cas {
let body =
canonical_body.map_err(|_| Error::other("pool metadata CAS body length cannot be represented"))?;
@@ -2880,48 +2915,44 @@ impl RemoteDisk {
} else {
attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
}
Ok(())
})();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_CANONICAL_BODY_DIGEST,
canonical_body_digest_started,
);
canonical_body_digest?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
prepare_started,
);
let rpc_started = rustfs_io_metrics::put_stage_timer();
let response = rustfs_io_metrics::observe_put_stage_future(
async {
if multipart_rename {
// Old peers must reject before an upload source can be changed.
client.rename_multipart_data(request).await
} else if pool_meta_cas {
client.rename_data_with_pool_meta_cas(request).await
} else if quota_fenced {
// The method itself proves quota-token enforcement; an old peer must fail closed.
client.rename_data_with_quota_fence(request).await
} else if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_CANONICAL_BODY_DIGEST,
canonical_body_digest_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
attempt_prepare_started,
);
let result = rustfs_io_metrics::observe_put_stage_future(
async {
if multipart_rename {
client.rename_multipart_data(request).await
} else if pool_meta_cas {
client.rename_data_with_pool_meta_cas(request).await
} else if quota_fenced {
client.rename_data_with_quota_fence(request).await
} else if incarnation_bound {
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
}
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await;
match result {
Ok(response) => break response,
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await
.map_err(|status| {
if (multipart_rename || pool_meta_cas) && status.code() == Code::Unimplemented {
DiskError::UnsupportedDisk
} else {
status.into()
Err(status) if (multipart_rename || pool_meta_cas) && status.code() == Code::Unimplemented => {
return Err(DiskError::UnsupportedDisk);
}
Err(status) => return Err(status.into()),
}
})?;
};
let response_process_started = rustfs_io_metrics::put_stage_timer();
let response_headers_started = rustfs_io_metrics::put_stage_timer();
let _ = response.metadata().len();
@@ -3756,34 +3787,45 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let file_info = compat_json(&fi)?;
let file_info_bin = encode_file_info_msgpack(&fi)?;
let file_info_bin = Bytes::from(encode_file_info_msgpack(&fi)?);
self.execute_with_timeout_for_op(
"write_metadata",
move || async move {
let disk = self.disk_ref().await;
let mut client = self.get_client().await?;
let mut request = Request::new(WriteMetadataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope(volume)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk,
volume: volume.to_string(),
path: path.to_string(),
file_info,
file_info_bin: file_info_bin.into(),
});
let canonical_body = rustfs_protos::canonical_write_metadata_request_body(request.get_ref());
let response = if request.get_ref().bucket_incarnation_id.is_empty() {
attach_mutation_body_digest(&mut request, canonical_body, "write_metadata")?;
client.write_metadata(request).await?
} else {
let body = canonical_body.map_err(|_| Error::other("write metadata request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?;
client.write_metadata_at_incarnation(request).await?
}
.into_inner();
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let file_info = compat_json(&fi, &policy)?;
let mut request = Request::new(WriteMetadataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope(volume)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
file_info,
file_info_bin: file_info_bin.clone(),
});
let canonical_body = rustfs_protos::canonical_write_metadata_request_body(request.get_ref());
policy.guard_request(&mut request);
let result = if request.get_ref().bucket_incarnation_id.is_empty() {
attach_mutation_body_digest(&mut request, canonical_body, "write_metadata")?;
client.write_metadata(request).await
} else {
let body =
canonical_body.map_err(|_| Error::other("write metadata request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?;
client.write_metadata_at_incarnation(request).await
};
match result {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -3837,29 +3879,38 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let file_info = compat_json(&fi)?;
let opts_str = compat_json(&opts)?;
let file_info_bin = encode_file_info_msgpack(&fi)?;
let opts_bin = encode_msgpack(opts)?;
let file_info_bin = Bytes::from(encode_file_info_msgpack(&fi)?);
let opts_bin = Bytes::from(encode_msgpack(opts)?);
self.execute_with_timeout_for_op(
"update_metadata",
move || async move {
let disk = self.disk_ref().await;
let mut client = self.get_client().await?;
let mut request = Request::new(UpdateMetadataRequest {
disk,
volume: volume.to_string(),
path: path.to_string(),
file_info,
opts: opts_str,
file_info_bin: file_info_bin.into(),
opts_bin: opts_bin.into(),
});
let canonical_body = rustfs_protos::canonical_update_metadata_request_body(request.get_ref());
attach_mutation_body_digest(&mut request, canonical_body, "update_metadata")?;
let response = client.update_metadata(request).await?.into_inner();
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let file_info = compat_json(&fi, &policy)?;
let opts_str = compat_json(opts, &policy)?;
let mut request = Request::new(UpdateMetadataRequest {
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
file_info,
opts: opts_str,
file_info_bin: file_info_bin.clone(),
opts_bin: opts_bin.clone(),
});
let canonical_body = rustfs_protos::canonical_update_metadata_request_body(request.get_ref());
policy.guard_request(&mut request);
attach_mutation_body_digest(&mut request, canonical_body, "update_metadata")?;
match client.update_metadata(request).await {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -3895,9 +3946,8 @@ impl DiskAPI for RemoteDisk {
);
let read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let encode_started = read_version_stage_timer(read_version_attribution_enabled);
let encoded_opts = compat_json(opts).and_then(|opts_str| encode_msgpack(opts).map(|opts_bin| (opts_str, opts_bin)));
let opts_bin = Bytes::from(encode_msgpack(opts)?);
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, encode_started);
let (opts_str, opts_bin) = encoded_opts?;
// Idempotent version read: eligible for the bounded transient-network retry so a single
// reset-by-peer during the read-after-write window does not erode the metadata read
@@ -3906,43 +3956,51 @@ impl DiskAPI for RemoteDisk {
self.execute_read_with_retry(
"read_version",
|| async {
let opts_str = opts_str.clone();
let opts_bin = opts_bin.clone();
let disk = self.disk_ref().await;
let mut client = self.get_client().await?;
let request_payload_bytes = read_version_attribution_enabled.then(|| {
disk.len()
.saturating_add(volume.len())
.saturating_add(path.len())
.saturating_add(version_id.len())
.saturating_add(opts_str.len())
.saturating_add(opts_bin.len())
});
let request = Request::new(ReadVersionRequest {
disk,
volume: volume.to_string(),
path: path.to_string(),
version_id: version_id.to_string(),
opts: opts_str,
opts_bin: opts_bin.into(),
});
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes);
}
let rpc_started = read_version_stage_timer(read_version_attribution_enabled);
let response = match client.read_version(request).await {
Ok(response) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
response.into_inner()
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error(
crate::cluster::rpc::runtime_sources::ReadVersionErrorStage::RpcStatus,
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let opts_str = compat_json(opts, &policy)?;
let request_payload_bytes = read_version_attribution_enabled.then(|| {
disk.len()
.saturating_add(volume.len())
.saturating_add(path.len())
.saturating_add(version_id.len())
.saturating_add(opts_str.len())
.saturating_add(opts_bin.len())
});
let mut request = Request::new(ReadVersionRequest {
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
version_id: version_id.to_string(),
opts: opts_str,
opts_bin: opts_bin.clone(),
});
policy.guard_request(&mut request);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(
request_payload_bytes,
);
return Err(err.into());
}
let rpc_started = read_version_stage_timer(read_version_attribution_enabled);
match client.read_version(request).await {
Ok(response) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
break response.into_inner();
}
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
retried_dual_write = true;
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error(
crate::cluster::rpc::runtime_sources::ReadVersionErrorStage::RpcStatus,
);
return Err(err.into());
}
}
};
@@ -4004,14 +4062,13 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let req = Arc::new(req);
let batch_read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let encode_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let batch_read_version_req = compat_json(&req)?;
let batch_read_version_req_bin = encode_msgpack(&req)?;
let batch_read_version_req_bin = Bytes::from(encode_msgpack(req.as_ref())?);
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started);
let request_payload_bytes = batch_read_version_attribution_enabled
.then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin));
let request_opts = req.opts;
let batch_rpc_req = Arc::clone(&req);
let request_opts = batch_rpc_req.opts;
let bulk_body = self.acquire_bulk_body().await?;
let batch_result = self
.execute_with_timeout_for_op(
@@ -4019,31 +4076,61 @@ impl DiskAPI for RemoteDisk {
move || async move {
let disk = self.disk_ref().await;
let disk_len = disk.len();
let mut client = self.get_bulk_client().await?;
let request = Request::new(BatchReadVersionRequest {
disk,
batch_read_version_req,
batch_read_version_req_bin: batch_read_version_req_bin.into(),
});
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
request_payload_bytes.saturating_add(disk_len),
);
}
let rpc_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let response_result =
measure_client_rpc(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, client.batch_read_version(request)).await;
let response = match response_result {
Ok(response) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
response.into_inner()
let (mut client, mut policy) = self.get_bulk_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let batch_read_version_req = compat_json(batch_rpc_req.as_ref(), &policy)?;
let request_payload_bytes = batch_read_version_attribution_enabled.then(|| {
batch_read_version_request_payload_len(
batch_rpc_req.as_ref(),
&batch_read_version_req,
&batch_read_version_req_bin,
)
});
let mut request = Request::new(BatchReadVersionRequest {
disk: disk.clone(),
batch_read_version_req,
batch_read_version_req_bin: batch_read_version_req_bin.clone(),
});
policy.guard_request(&mut request);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
request_payload_bytes.saturating_add(disk_len),
);
}
Err(status) if status.code() == Code::Unimplemented => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
if mode.should_fallback_on_unimplemented() {
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_FALLBACK_UNIMPLEMENTED);
let rpc_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let response_result =
measure_client_rpc(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, client.batch_read_version(request))
.await;
match response_result {
Ok(response) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
break response.into_inner();
}
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
retried_dual_write = true;
}
Err(status) if status.code() == Code::Unimplemented => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
if mode.should_fallback_on_unimplemented() {
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_FALLBACK_UNIMPLEMENTED);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "fallback_unimplemented",
"Remote disk BatchReadVersion unsupported; falling back to unary read_version"
);
return Ok(None);
}
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_UNSUPPORTED_NO_FALLBACK);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
@@ -4051,30 +4138,16 @@ impl DiskAPI for RemoteDisk {
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "fallback_unimplemented",
"Remote disk BatchReadVersion unsupported; falling back to unary read_version"
state = "unsupported_no_fallback",
"Remote disk BatchReadVersion unsupported and explicit batch RPC mode forbids fallback"
);
return Ok(None);
return Err(Error::from(status));
}
Err(status) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
return Err(Error::from(status));
}
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_UNSUPPORTED_NO_FALLBACK);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "unsupported_no_fallback",
"Remote disk BatchReadVersion unsupported and explicit batch RPC mode forbids fallback"
);
return Err(Error::from(status));
}
Err(status) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
return Err(Error::from(status));
}
};
@@ -4110,6 +4183,7 @@ impl DiskAPI for RemoteDisk {
// instead of racing the whole batch against one drive timeout.
None => {
drop(bulk_body);
let req = Arc::try_unwrap(req).unwrap_or_else(|req| (*req).clone());
batch_read_version_one_by_one(self, req).await
}
}
@@ -5056,18 +5130,27 @@ impl DiskAPI for RemoteDisk {
let _bulk_body = self.acquire_bulk_body().await?;
self.execute_with_timeout(
|| async {
let read_multiple_req = compat_json(&req)?;
let read_multiple_req_bin = encode_msgpack(&req)?;
let read_multiple_req_bin = Bytes::from(encode_msgpack(&req)?);
let disk = self.disk_ref().await;
let mut client = self.get_bulk_client().await?;
let request = Request::new(ReadMultipleRequest {
disk,
read_multiple_req,
read_multiple_req_bin: read_multiple_req_bin.into(),
});
let response = measure_client_rpc(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, client.read_multiple(request)).await;
let response = response?.into_inner();
let (mut client, mut policy) = self.get_bulk_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let read_multiple_req = compat_json(&req, &policy)?;
let mut request = Request::new(ReadMultipleRequest {
disk: disk.clone(),
read_multiple_req,
read_multiple_req_bin: read_multiple_req_bin.clone(),
});
policy.guard_request(&mut request);
let result = measure_client_rpc(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, client.read_multiple(request)).await;
match result {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -6254,6 +6337,40 @@ mod tests {
}
}
#[test]
fn msgpack_eligible_requests_bind_one_policy_and_bounded_local_retry() {
let source = rustfs_protos::compat_manifest::production_source(include_str!("remote_disk.rs"), "remote_disk.rs");
for function in [
"rename_data_borrowed_with_fence",
"write_metadata",
"update_metadata",
"read_version",
"batch_read_version",
"read_multiple",
] {
let start = source
.find(&format!("async fn {function}("))
.unwrap_or_else(|| panic!("missing {function}"));
let tail = &source[start..];
let next_instrument = ["\n #[tracing::", "instrument"].concat();
let end = tail[1..].find(&next_instrument).map_or(tail.len(), |offset| offset + 1);
let body = &tail[..end];
assert!(body.contains("with_msgpack_policy().await"), "{function} must capture one RPC policy");
assert!(
body.contains("policy.guard_request(&mut request)"),
"{function} must bind the generation guard"
);
assert!(
body.contains("policy.retry_dual_write_after_local_rejection()"),
"{function} must allow only the bounded local dual-write retry"
);
assert!(
body.contains("let mut retried_dual_write = false;"),
"{function} must bound retry to one attempt"
);
}
}
#[test]
fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
@@ -7162,7 +7279,8 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let json = compat_json(&resp).expect("compat_json should encode");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
assert!(!json.is_empty());
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
},
@@ -7187,7 +7305,8 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let json = compat_json(&resp).expect("compat_json should encode");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
assert!(!json.is_empty(), "old JSON peers must remain compatible without fleet confirmation");
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
@@ -7204,7 +7323,8 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let json = compat_json(&resp).expect("compat_json should encode");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
assert!(json.is_empty(), "msgpack-only may empty JSON only after explicit fleet confirmation");
},
@@ -7220,7 +7340,8 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let json = compat_json(&resp).expect("compat_json should encode");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
assert!(json.is_empty(), "both gates should enter msgpack-only send mode");
},
@@ -7238,7 +7359,8 @@ mod tests {
] {
with_internode_msgpack_env(vars, || {
let resp = sample_read_multiple_resp("file", b"data");
let json = compat_json(&resp).expect("compat_json should encode");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
assert!(!json.is_empty(), "removing either gate should restore old-peer JSON compatibility");
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
+21 -119
View File
@@ -1463,36 +1463,22 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
let observed = load_observed_data_usage_snapshot(store.clone()).await.ok().flatten();
let (selected, selected_is_current_format) =
select_admin_data_usage_snapshot(authoritative, source.is_authoritative(), observed);
Ok(normalize_loaded_data_usage(selected, selected_is_current_format).await.0)
}
async fn project_admin_data_usage_for_current_buckets(
store: Arc<ECStore>,
mut response: DataUsageInfo,
with_memory_overlay: bool,
) -> DataUsageInfo {
let mut response = normalize_loaded_data_usage(selected, selected_is_current_format).await.0;
use futures::StreamExt as _;
let snapshot_measured = response.is_complete_bucket_usage_snapshot() || response.is_valid_partial_snapshot();
let overlay_authoritative = with_memory_overlay && !runtime_sources::setup_is_dist_erasure().await;
let deadline = tokio::time::Instant::now() + ADMIN_USAGE_INCARNATION_TIMEOUT;
let mut incarnations = HashMap::new();
{
// Identity reads happen after releasing snapshot and refresh locks,
// including on cache hits. Metadata transaction reads linearize each
// bucket against creation/deletion without relying on peer notices.
// The common deadline cancels unfinished work and bounds added latency.
let mut buckets = response
.buckets_usage
// Display-only identity reads must not serialize every bucket behind a
// slow metadata lock while holding the shared admin refresh permit.
// The common deadline bounds their total added latency; dropping this
// stream cancels all unfinished reads without spawning detached work.
// Own the names and store across awaits so this future remains Send
// when composed into spawned metrics collectors.
let buckets = response
.usage_snapshot_bucket_observations
.keys()
.chain(response.usage_snapshot_bucket_observations.keys())
.cloned()
.collect::<HashSet<_>>();
if overlay_authoritative {
// Copy only names while holding the memory lock, then release it
// before storage I/O. A later new entry has no identity proof and
// will remain unknown in this response.
buckets.extend(memory_cache().read().await.keys().cloned());
}
.collect::<Vec<_>>();
let mut identities = futures::stream::iter(buckets)
.map(move |bucket| {
let store = store.clone();
@@ -1506,6 +1492,7 @@ async fn project_admin_data_usage_for_current_buckets(
let Ok(Some((bucket, identity))) = tokio::time::timeout_at(deadline, identities.next()).await else {
break;
};
// Unavailable, timed-out or recreated buckets retain canonical values.
if let Ok(incarnation) = identity
&& !incarnation.is_nil()
{
@@ -1514,20 +1501,7 @@ async fn project_admin_data_usage_for_current_buckets(
}
}
response.project_bucket_observations_for_admin(&incarnations);
if with_memory_overlay
&& apply_bucket_usage_memory_overlay_if_authoritative(&mut response, overlay_authoritative).await
&& snapshot_measured
{
response.usage_snapshot_complete = false;
response.usage_snapshot_partial = true;
response.usage_snapshot_converged = Some(false);
response.usage_snapshot_admin_projection = true;
response.replication_info.clear();
response.tier_stats = None;
response.unknown_tier_stats = None;
}
response.retain_current_bucket_usage_for_admin(&incarnations);
response
Ok(response)
}
fn discard_incomplete_bucket_usage(data_usage_info: &mut DataUsageInfo) {
@@ -1716,7 +1690,11 @@ async fn project_data_usage_from_backend_cached<T>(
}
}
async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
/// Load the freshest structurally complete snapshot for authenticated admin
/// observability. A scan raced by namespace activity may be selected here, but
/// never by [`load_data_usage_from_backend_cached`], which remains the
/// converged source for quota admission.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
loop {
@@ -1765,38 +1743,6 @@ async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<Da
}
}
/// Load cached scanner statistics for authenticated admin observability.
///
/// Each response verifies bucket identities against current storage metadata,
/// including cache hits. Missing, failed, or timed-out identity reads leave that
/// bucket's usage absent (unknown). Legacy values without an incarnation stay
/// unknown until a scanner snapshot certifies their identity. Per-bucket reads
/// linearize independently; this is not an atomic namespace snapshot.
/// Identity reads have a shared deadline and do not outlive cancellation. The
/// unchanged cached baseline remains independent of quota and persistence.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, false).await)
}
/// Load the admin view with the existing authoritative single-process overlay.
///
/// The overlay is applied before current-identity projection so it cannot
/// reintroduce statistics from a retired or unverified bucket generation.
/// Distributed nodes continue to use scanner statistics without local totals.
/// Cancellation, unknown values, and per-bucket read semantics are the same as
/// [`load_admin_data_usage_from_backend_cached`]. No overlay is persisted.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_with_memory_overlay(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, true).await)
}
/// Invalidate the process-local persisted usage snapshot cache after a durable
/// scanner write.
pub async fn invalidate_data_usage_snapshot_cache() {
@@ -2859,17 +2805,16 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
*cache = next_cache;
}
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) -> bool {
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) {
if !authoritative {
return false;
return;
}
let cache = memory_cache().read().await;
if cache.is_empty() {
return false;
return;
}
let was_complete = data_usage_info.usage_snapshot_complete;
let persisted_update = data_usage_info.last_update;
let mut changed = false;
let mut added_bucket = false;
@@ -2906,7 +2851,6 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
data_usage_info.calculate_totals();
}
changed || was_complete != data_usage_info.usage_snapshot_complete
}
pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
@@ -3800,23 +3744,6 @@ mod tests {
*cache_updating().write().await = false;
}
async fn bind_admin_usage_test_identities(store: &Arc<ECStore>, snapshot: &mut DataUsageInfo) {
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
for (bucket, usage) in &mut snapshot.buckets_usage {
store
.make_bucket(bucket, &crate::storage_api_contracts::bucket::MakeBucketOptions::default())
.await
.expect("create a real bucket for current admin identity validation");
usage.bucket_incarnation = Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("read the storage-owned fixture identity")
.to_string(),
);
}
}
fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
let mut info = DataUsageInfo {
last_update: Some(last_update),
@@ -4352,11 +4279,7 @@ mod tests {
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let (mut authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
bind_admin_usage_test_identities(&store, &mut authoritative).await;
for (bucket, usage) in &mut observed.buckets_usage {
usage.bucket_incarnation = authoritative.buckets_usage[bucket].bucket_incarnation.clone();
}
let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
@@ -4405,11 +4328,6 @@ mod tests {
reconciled.scanner_epoch = Some(4);
reconciled.scanner_cycle = Some(12);
reconciled.usage_snapshot_converged = Some(true);
reconciled
.buckets_usage
.get_mut("large-bucket")
.expect("reconciled bucket")
.bucket_incarnation = authoritative.buckets_usage["large-bucket"].bucket_incarnation.clone();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4931,12 +4849,6 @@ mod tests {
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut current).await;
backup
.buckets_usage
.get_mut("existing")
.expect("backup bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
@@ -4959,11 +4871,6 @@ mod tests {
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
after_delete
.buckets_usage
.get_mut("existing")
.expect("post-delete bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
@@ -5025,7 +4932,6 @@ mod tests {
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut authoritative).await;
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -5040,10 +4946,6 @@ mod tests {
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info.buckets_usage
.get_mut("bucket")
.expect("observed bucket")
.bucket_incarnation = authoritative.buckets_usage["bucket"].bucket_incarnation.clone();
info
};
let first = observation(7, 11, 200, 11, 1_100);
@@ -221,7 +221,7 @@ async fn companion_admin_cache_isolated_from_quota_memory_and_persistence() {
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_buckets() {
async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buckets() {
use crate::bucket::metadata_sys::{
acquire_metadata_transaction_guard, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
@@ -280,11 +280,8 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
.await
.expect("optional projection must return before the blocked metadata lock's long timeout")
.expect("canonical admin response stays available");
assert!(
!response.buckets_usage.contains_key(blocked),
"an unavailable current identity cannot bind the canonical value"
);
assert!(!response.bucket_sizes.contains_key(blocked));
assert_eq!(response.buckets_usage[blocked].size, 100);
assert_eq!(response.buckets_usage[blocked].usage_value_state, Some(BucketUsageValueState::Carried));
assert_eq!(response.buckets_usage[healthy].size, 250);
assert_eq!(response.buckets_usage[healthy].usage_value_state, Some(BucketUsageValueState::Observed));
assert!(response.usage_snapshot_admin_projection);
@@ -296,12 +293,13 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
.is_ok()
);
drop(blocked_guard);
invalidate_data_usage_snapshot_cache().await;
let recovered = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh after lock recovery");
assert_eq!(
recovered.buckets_usage[blocked].size, 250,
"unavailable identity can recover on a cache hit"
"unavailable identity can recover on the next refresh"
);
assert_eq!(recovered.buckets_usage[healthy].size, 250);
let quota = load_data_usage_from_backend_cached(store)
@@ -310,201 +308,3 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
assert_eq!(quota.buckets_usage[blocked].size, 100);
assert_eq!(quota.buckets_usage[healthy].size, 100);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn cached_canonical_admin_usage_rechecks_current_identity_without_peer_invalidation() {
use crate::bucket::metadata::{load_bucket_incarnation, save_bucket_incarnation};
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-cache";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
canonical.usage_snapshot_pending_buckets.clear();
canonical.usage_snapshot_pending_bucket_count = 0;
canonical
.buckets_usage
.get_mut(bucket)
.expect("canonical bucket")
.objects_count = 107;
canonical.calculate_totals();
let original_bytes = serde_json::to_vec(&canonical).expect("encode canonical old bucket");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), original_bytes.clone())
.await
.expect("persist canonical root");
invalidate_data_usage_snapshot_cache().await;
let first = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("prime raw admin cache");
assert_eq!(first.buckets_usage[bucket].objects_count, 107);
assert!(first.is_complete_bucket_usage_snapshot());
let cache_generation = admin_data_usage_snapshot_generation();
let new_identity = uuid::Uuid::new_v4();
// Simulate the storage-owned identity changing on another node while this
// process has not received its invalidation notice. Do not clear the cache.
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish changed disk identity");
assert_eq!(
load_bucket_incarnation(store.clone(), bucket)
.await
.expect("read new sidecar"),
Some(new_identity)
);
let second = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("read raw cache with fresh identity");
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
assert!(
!second.buckets_usage.contains_key(bucket),
"same name cannot retain the predecessor's 107"
);
assert!(!second.bucket_sizes.contains_key(bucket));
assert_eq!(second.usage_snapshot_converged, Some(false));
assert!(second.usage_snapshot_admin_projection);
assert!(reject_admin_usage_projection(&second).is_err());
assert!(certified_bucket_usage(&second, bucket, new_identity).is_none());
assert!(parse_usage_snapshot(&serde_json::to_vec(&second).expect("encode display response")).is_err());
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("independent quota baseline");
assert_eq!(quota.buckets_usage[bucket].objects_count, 107);
assert_eq!(
read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("unchanged persisted root"),
original_bytes
);
// Rebinding a cached raw baseline is transient: recovery does not need to
// reload or permanently blank a healthy bucket's statistics.
save_bucket_incarnation(store.clone(), bucket, old_identity)
.await
.expect("restore fixture identity");
let recovered = load_admin_data_usage_from_backend_cached(store)
.await
.expect("recover on another cache hit");
assert_eq!(recovered.buckets_usage[bucket].objects_count, 107);
assert!(recovered.is_complete_bucket_usage_snapshot());
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_cannot_reintroduce_a_predecessor_identity() {
use crate::bucket::metadata::save_bucket_incarnation;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-overlay";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode canonical root"),
)
.await
.expect("persist overlay fixture");
invalidate_data_usage_snapshot_cache().await;
let cached_usage = canonical.buckets_usage[bucket].clone();
replace_bucket_usage_memory_from_authoritative(bucket, cached_usage, SystemTime::now()).await;
let new_identity = uuid::Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish a replacement identity");
let response = load_admin_data_usage_with_memory_overlay(store.clone())
.await
.expect("project after memory overlay");
assert!(!response.buckets_usage.contains_key(bucket));
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(response.usage_snapshot_admin_projection);
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_overlay_flags_only_change_keeps_identity_bound_cold_usage() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(
!runtime_sources::setup_is_dist_erasure().await,
"exercise the production single-process overlay"
);
let cold = "usage-overlay-cold";
let unverified = "usage-overlay-unverified";
for bucket in [cold, unverified] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay fixture bucket");
}
let identity = store
.bucket_incarnation_id_from_disk(cold)
.await
.expect("cold bucket identity");
let mut canonical = companion_snapshot(cold, identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode cold canonical value"),
)
.await
.expect("persist cold baseline");
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
memory_cache().write().await.insert(
unverified.to_string(),
cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false),
);
record_bucket_object_write_memory(unverified, None, 42).await;
{
let cache = memory_cache().read().await;
assert!(cache[unverified].dirty);
assert!(!cache[unverified].authoritative);
}
let response = load_admin_data_usage_with_memory_overlay(store)
.await
.expect("identity-filtered flags-only overlay");
assert_eq!(response.buckets_usage[cold].size, 100);
assert_eq!(response.objects_total_size, 100);
assert_eq!(response.bucket_sizes[cold], 100);
assert!(!response.buckets_usage.contains_key(unverified));
assert!(response.usage_snapshot_admin_projection);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
@@ -1033,26 +1033,16 @@ mod tests {
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket("bucket", &MakeBucketOptions::default())
.await
.expect("create a real observed bucket");
let incarnation = store
.bucket_incarnation_id_from_disk("bucket")
.await
.expect("read current storage-owned incarnation");
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
@@ -1076,7 +1066,6 @@ mod tests {
versions_count: 4,
delete_markers_count: 1,
size: 6,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
},
)]
@@ -1128,46 +1117,6 @@ mod tests {
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn identity_filtered_usage_remains_unknown_in_server_info() {
let mut info = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
buckets_count: 1,
objects_total_count: 107,
objects_total_size: 1024,
buckets_usage: [(
"bucket".to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 107,
size: 1024,
bucket_incarnation: Some("11111111-1111-4111-8111-111111111111".to_string()),
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
..Default::default()
};
let current =
std::collections::HashMap::from([("bucket".to_string(), "22222222-2222-4222-8222-222222222222".to_string())]);
assert_eq!(info.retain_current_bucket_usage_for_admin(&current), 1);
assert!(info.buckets_usage.is_empty());
assert!(info.usage_snapshot_admin_projection);
assert_eq!(info.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
+2 -7
View File
@@ -1771,7 +1771,7 @@ fn run_file_fdatasync_group_worker(group: Arc<FileFdatasyncGroup>) -> impl std::
#[cfg(test)]
file_sync_probe::record_group_batch(batch_file_count);
rustfs_io_metrics::record_put_rename_fdatasync_batch(
rustfs_io_metrics::PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL,
rustfs_io_metrics::PUT_RENAME_FDATASYNC_BATCH_MODE_GROUPED,
batch_file_count,
);
let result = if let Some(disk_permits) = group.disk_permits.upgrade() {
@@ -12461,7 +12461,6 @@ mod tests {
let _group_commit = set_file_fdatasync_group_commit_for_test(true);
set_file_fdatasync_group_commit_wait_for_test(0);
file_fdatasync_group_commit_override::set_follower_timeout_micros(1_000);
clear_file_fdatasync_group_commit_for_test();
let temp_dir = tempdir().expect("create temp dir");
let first_dir = temp_dir.path().join("first");
@@ -12489,7 +12488,7 @@ mod tests {
let second_limiter = limiter.clone();
let second_path = second_dir.clone();
let mut second = tokio::spawn(async move { sync_dir_files_with_limiter(second_path, second_limiter).await });
let second = tokio::spawn(async move { sync_dir_files_with_limiter(second_path, second_limiter).await });
tokio::time::timeout(Duration::from_secs(30), async {
loop {
if file_fdatasync_group_commit_counts_for_test().1 == 2 {
@@ -12502,10 +12501,6 @@ mod tests {
.expect("second waiter should enqueue before releasing the grouped batch");
release_batch_tx.send(()).expect("release group batch hook");
file_sync_probe::wait_for_active(1).await;
assert!(
tokio::time::timeout(Duration::from_millis(10), &mut second).await.is_err(),
"a claimed follower waits for its batch even after its queue deadline"
);
assert_eq!(file_sync_probe::attempts(), 1, "claimed files must not enter an independent sync");
assert_eq!(
+4
View File
@@ -1388,6 +1388,10 @@ pub struct ObjectOptions {
#[doc(hidden)]
pub multipart_completion_authorization:
Option<Arc<dyn crate::storage_api_contracts::multipart::MultipartCompletionAuthorization<Error = Error>>>,
/// In-process authorization against the selected upload, while Abort holds its write lock.
#[doc(hidden)]
pub multipart_abort_authorization:
Option<Arc<dyn crate::storage_api_contracts::multipart::MultipartAbortAuthorization<Error = Error>>>,
pub eval_metadata: Option<HashMap<String, String>>,
/// Internal compare-and-set condition for replication workers publishing
/// terminal status after remote I/O. Storage validates it while holding
@@ -83,6 +83,7 @@ use crate::set_disk::core::multipart_source::{
use crate::set_disk::mem;
use crate::set_disk::metadata_sys;
use crate::set_disk::runtime_sources;
use crate::storage_api_contracts::multipart::ListPartsOwner;
#[cfg(test)]
use crate::storage_api_contracts::multipart::MultipartOperations;
#[cfg(test)]
@@ -683,6 +684,22 @@ fn fence_commit_on_lock_loss(guard: Option<&ObjectLockDiagGuard>, mode: &'static
Ok(())
}
fn recorded_multipart_owner(
metadata: &HashMap<String, String>,
bucket: &str,
encoded_object: &str,
) -> Option<crate::storage_api_contracts::multipart::MultipartOwner> {
let recorded_bucket = get_consistent_str(metadata, "multipart-bucket")?;
let recorded_object = get_consistent_str(metadata, "multipart-object")?;
if recorded_bucket != bucket || recorded_object != encoded_object {
return None;
}
Some(crate::storage_api_contracts::multipart::MultipartOwner {
bucket: recorded_bucket.to_owned(),
object: decode_dir_object(recorded_object),
})
}
fn multipart_bucket_incarnation_id(metadata: &HashMap<String, String>) -> Result<Option<Uuid>> {
let Some(value) = rustfs_utils::http::metadata_compat::get_consistent_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) else {
if rustfs_utils::http::metadata_compat::contains_key_str(metadata, SUFFIX_BUCKET_INCARNATION_ID) {
@@ -3190,7 +3207,21 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.cloned()
.unwrap_or_else(|| storageclass::STANDARD.to_string());
// Use the same metadata snapshot that produced the page.
let recorded_owner = match (
get_consistent_str(&fi.metadata, "multipart-bucket"),
get_consistent_str(&fi.metadata, "multipart-object"),
) {
(Some(recorded_bucket), Some(recorded_object)) if recorded_bucket == bucket && recorded_object == object => {
Some(ListPartsOwner {
bucket: recorded_bucket.to_owned(),
object: decode_dir_object(recorded_object),
})
}
_ => None,
};
let mut ret = ListPartsInfo {
recorded_owner,
bucket: bucket.to_owned(),
object: decode_dir_object(object),
upload_id: upload_id.to_owned(),
@@ -3597,6 +3628,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
ensure_multipart_bucket_incarnation(&self.ctx, &fi, bucket, object, upload_id, opts.expected_bucket_incarnation_id)
.await?;
ensure_multipart_bucket_lifecycle_lock_held(bucket, object, opts)?;
if let Some(authorization) = &opts.multipart_abort_authorization {
let owner = recorded_multipart_owner(&fi.metadata, bucket, object);
authorization.authorize(owner.as_ref())?;
}
let disks = self.get_disks_internal().await;
let cleanup = prepare_source_cleanup(
&disks,
@@ -6434,6 +6469,8 @@ mod tests {
assert!(empty.parts.is_empty());
assert!(!empty.is_truncated);
assert_eq!(empty.next_part_number_marker, None);
let owner = empty.recorded_owner.expect("the empty page has the recorded owner");
assert_eq!((owner.bucket.as_str(), owner.object.as_str()), (bucket, object));
for part_number in [10, 1, 3] {
put_test_part(&set_disks, bucket, object, &upload.upload_id, part_number, b"part", 4).await;
@@ -6460,6 +6497,66 @@ mod tests {
assert_eq!(page.parts.iter().map(|part| part.part_num).collect::<Vec<_>>(), expected_parts);
assert_eq!(page.is_truncated, next_marker.is_some());
assert_eq!(page.next_part_number_marker, next_marker);
assert_eq!(page.recorded_owner.as_ref(), Some(&owner));
}
}
#[tokio::test]
async fn list_parts_baseline_legacy_owner_metadata() {
let (_roots, disks, set) = hermetic_set_disks(4).await;
let bucket = "list-parts-owner-observation";
let object = "object";
let opts = ObjectOptions::default();
make_bucket_on_all(&disks, bucket).await;
let upload = set.new_multipart_upload(bucket, object, &opts).await.expect("real upload");
let path = SetDisks::get_upload_id_dir(bucket, object, &upload.upload_id);
let owner_keys = [
RUSTFS_MULTIPART_BUCKET_KEY,
"x-minio-internal-multipart-bucket",
RUSTFS_MULTIPART_OBJECT_KEY,
"x-minio-internal-multipart-object",
];
for (label, values) in [
("rustfs", [Some(bucket), None, Some(object), None]),
("minio", [None, Some(bucket), None, Some(object)]),
("both", [Some(bucket), Some(bucket), Some(object), Some(object)]),
("no bucket", [None, None, Some(object), None]),
("no object", [Some(bucket), None, None, None]),
("other bucket", [Some("other-bucket"), None, Some(object), None]),
("other object", [Some(bucket), None, Some("other-object"), None]),
("bucket aliases differ", [Some(bucket), Some("other-bucket"), Some(object), None]),
("object aliases differ", [Some(bucket), None, Some(object), Some("other-object")]),
] {
for disk in &disks {
let mut fi = disk
.read_version("", RUSTFS_META_MULTIPART_BUCKET, &path, "", &ReadOptions::default())
.await
.expect("read actual upload metadata");
for (key, value) in owner_keys.into_iter().zip(values) {
if let Some(value) = value {
fi.metadata.insert(key.to_owned(), value.to_owned());
} else {
fi.metadata.remove(key);
}
}
disk.write_metadata(bucket, RUSTFS_META_MULTIPART_BUCKET, &path, fi)
.await
.expect("write the synthetic owner metadata");
}
let page = set
.list_object_parts(bucket, object, &upload.upload_id, None, 1, &opts)
.await
.unwrap_or_else(|error| panic!("{label}: {error:?}"));
assert_eq!((page.bucket.as_str(), page.object.as_str()), (bucket, object), "{label}");
assert!(page.parts.is_empty(), "{label}");
let expected = matches!(label, "rustfs" | "minio" | "both").then_some((bucket, object));
assert_eq!(
page.recorded_owner
.as_ref()
.map(|owner| (owner.bucket.as_str(), owner.object.as_str())),
expected,
"{label}"
);
}
}
@@ -1029,3 +1029,229 @@ mod native_abort_closure_tests {
include!("multipart_initial_probe_tests.rs");
}
}
mod abort_owner_tests {
use super::*;
use crate::storage_api_contracts::multipart::{MultipartAbortAuthorization, MultipartOwner};
#[test]
fn recorded_multipart_owner_requires_unambiguous_metadata_for_the_selected_namespace() {
const BUCKET: &str = "abort-owner";
const ENCODED: &str = "directory__XLDIR__";
const RUSTFS: &str = RUSTFS_MULTIPART_BUCKET_KEY;
const MINIO: &str = "x-minio-internal-multipart-bucket";
const RUSTFS_OBJECT: &str = RUSTFS_MULTIPART_OBJECT_KEY;
const MINIO_OBJECT: &str = "x-minio-internal-multipart-object";
let expected = MultipartOwner {
bucket: BUCKET.to_owned(),
object: "directory/".to_owned(),
};
let rustfs = [(RUSTFS, BUCKET), (RUSTFS_OBJECT, ENCODED)];
let minio = [(MINIO, BUCKET), (MINIO_OBJECT, ENCODED)];
for fields in [rustfs.to_vec(), minio.to_vec(), [rustfs, minio].concat()] {
let metadata = fields
.into_iter()
.map(|(key, value)| (key.to_owned(), value.to_owned()))
.collect();
assert_eq!(recorded_multipart_owner(&metadata, BUCKET, ENCODED), Some(expected.clone()));
}
for fields in [
vec![(RUSTFS, BUCKET)],
vec![(RUSTFS_OBJECT, ENCODED)],
vec![(RUSTFS, ""), (RUSTFS_OBJECT, ENCODED)],
vec![(RUSTFS, BUCKET), (RUSTFS_OBJECT, "")],
vec![(RUSTFS, "foreign"), (RUSTFS_OBJECT, ENCODED)],
vec![(RUSTFS, BUCKET), (RUSTFS_OBJECT, "directory/")],
[rustfs, [(MINIO, "foreign"), (MINIO_OBJECT, ENCODED)]].concat(),
[rustfs, [(MINIO, BUCKET), ("X-MINIO-INTERNAL-MULTIPART-OBJECT", "other")]].concat(),
] {
let metadata = fields
.iter()
.map(|(key, value)| (key.to_string(), value.to_string()))
.collect();
assert_eq!(recorded_multipart_owner(&metadata, BUCKET, ENCODED), None, "{fields:?}");
}
}
struct AbortGate {
expected: Option<MultipartOwner>,
accepted: bool,
calls: AtomicUsize,
}
impl MultipartAbortAuthorization for AbortGate {
type Error = StorageError;
fn authorize(&self, owner: Option<&MultipartOwner>) -> Result<()> {
self.calls.fetch_add(1, Ordering::Relaxed);
assert_eq!(owner, self.expected.as_ref(), "gate sees the actual selected upload owner");
if self.accepted {
Ok(())
} else {
Err(StorageError::InvalidUploadID(String::new(), String::new(), String::new()))
}
}
}
#[tokio::test]
async fn abort_owner_callback_controls_physical_deletion() {
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "abort-locked-owner";
let key = "staged-object";
make_bucket_on_all(&disks, bucket).await;
let (id, _) = stage_upload_with_create_opts(&set, bucket, key, b"staged part", &ObjectOptions::default()).await;
let path = SetDisks::get_upload_id_dir(bucket, key, &id);
let (fi, _) = set
.check_upload_id_exists(bucket, key, &id, true)
.await
.expect("staged upload");
let part = format!("{path}/{}/part.1", fi.data_dir.expect("staged data directory"));
let expected = MultipartOwner {
bucket: bucket.to_owned(),
object: key.to_owned(),
};
let mut before = Vec::new();
for disk in &disks {
before.push(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &part)
.await
.expect("staged shard"),
);
}
let rejected = Arc::new(AbortGate {
expected: Some(expected.clone()),
accepted: false,
calls: AtomicUsize::new(0),
});
let opts = ObjectOptions {
multipart_abort_authorization: Some(rejected.clone()),
..Default::default()
};
let result = set.abort_multipart_upload(bucket, key, &id, &opts).await;
for (disk, staged) in disks.iter().zip(before) {
let after = disk
.read_all(RUSTFS_META_MULTIPART_BUCKET, &part)
.await
.expect("denied Abort preserves the shard");
assert_eq!(staged, after);
}
assert!(matches!(result, Err(StorageError::InvalidUploadID(..))));
assert_eq!(rejected.calls.load(Ordering::Relaxed), 1);
let allowed = Arc::new(AbortGate {
expected: Some(expected),
accepted: true,
calls: AtomicUsize::new(0),
});
let opts = ObjectOptions {
multipart_abort_authorization: Some(allowed.clone()),
..Default::default()
};
set.abort_multipart_upload(bucket, key, &id, &opts)
.await
.expect("owner permits Abort");
assert_eq!(allowed.calls.load(Ordering::Relaxed), 1);
for disk in &disks {
assert!(matches!(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, &part).await,
Err(DiskError::FileNotFound)
));
}
}
#[tokio::test]
async fn abort_owner_comes_from_the_selected_metadata_on_all_disks() {
use rustfs_utils::http::{get_str, remove_str};
let (_dirs, disks, set) = hermetic_set_disks(4).await;
let bucket = "abort-owner-metadata";
make_bucket_on_all(&disks, bucket).await;
for (key, bucket_fields, object_fields, accepted) in [
("rustfs", [Some(bucket), None], [Some("rustfs"), None], true),
("minio", [None, Some(bucket)], [None, Some("minio")], true),
("matching", [Some(bucket), Some(bucket)], [Some("matching"), Some("matching")], true),
("empty", [Some(bucket), Some(bucket)], [Some("empty"), Some("")], false),
("missing", [None, None], [Some("missing"), None], false),
("foreign-bucket", [Some("foreign"), None], [Some("foreign-bucket"), None], false),
("foreign-key", [Some(bucket), None], [Some("unrelated"), None], false),
("conflict", [Some(bucket), Some("foreign")], [Some("conflict"), Some("conflict")], false),
] {
let (id, _) = stage_upload_with_create_opts(&set, bucket, key, b"staged shard data", &ObjectOptions::default()).await;
let path = SetDisks::get_upload_id_dir(bucket, key, &id);
pin_historical_unmarked_upload(&disks, bucket, &path).await;
let mut part_path = None;
let mut staged = Vec::new();
let fields = [
(RUSTFS_MULTIPART_BUCKET_KEY, bucket_fields[0]),
("x-minio-internal-multipart-bucket", bucket_fields[1]),
(RUSTFS_MULTIPART_OBJECT_KEY, object_fields[0]),
("x-minio-internal-multipart-object", object_fields[1]),
];
for disk in &disks {
let mut head = disk
.read_version("", RUSTFS_META_MULTIPART_BUCKET, &path, "", &ReadOptions::default())
.await
.expect("read the staged metadata on each disk");
let part = part_path.get_or_insert_with(|| format!("{path}/{}/part.1", head.data_dir.expect("part directory")));
staged.push(
disk.read_all(RUSTFS_META_MULTIPART_BUCKET, part)
.await
.expect("read staged part before Abort"),
);
assert_eq!(get_str(&head.metadata, "multipart-bucket").as_deref(), Some(bucket));
remove_str(&mut head.metadata, "multipart-bucket");
remove_str(&mut head.metadata, "multipart-object");
for (name, value) in fields {
if let Some(value) = value {
head.metadata.insert(name.to_owned(), value.to_owned());
}
}
disk.write_metadata(bucket, RUSTFS_META_MULTIPART_BUCKET, &path, head)
.await
.expect("write the test owner on each disk");
let persisted = disk
.read_version("", RUSTFS_META_MULTIPART_BUCKET, &path, "", &ReadOptions::default())
.await
.expect("verify the actual persisted metadata");
for (name, value) in fields {
assert_eq!(persisted.metadata.get(name).map(String::as_str), value);
}
}
let gate = Arc::new(AbortGate {
expected: accepted.then(|| MultipartOwner {
bucket: bucket.to_owned(),
object: key.to_owned(),
}),
accepted,
calls: AtomicUsize::new(0),
});
let opts = ObjectOptions {
multipart_abort_authorization: Some(gate.clone()),
..Default::default()
};
let result = set.abort_multipart_upload(bucket, key, &id, &opts).await;
for (disk, previous) in disks.iter().zip(staged) {
let after = disk
.read_all(RUSTFS_META_MULTIPART_BUCKET, part_path.as_deref().expect("part path"))
.await;
if accepted {
assert!(matches!(after, Err(DiskError::FileNotFound)), "{key}: authorized data removed");
} else {
assert_eq!(after.expect("refused Abort preserves the staged part"), previous, "{key}");
}
}
let calls = gate.calls.load(Ordering::Relaxed);
if accepted {
assert_eq!(calls, 1, "{key}");
result.expect("Abort with agreeing persisted owner");
} else {
assert!(result.is_err(), "{key}: invalid metadata must not authorize Abort");
if key == "empty" {
// The existing source inspection refuses this head before selecting an owner.
assert_eq!(calls, 0, "malformed physical owner was refused before the callback");
}
}
}
}
}
@@ -52,9 +52,10 @@ pub(crate) mod list {
}
pub(crate) mod multipart {
pub(crate) use rustfs_storage_api::multipart::ListPartsOwner;
pub(crate) use rustfs_storage_api::{
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartCompletionAuthorization, MultipartInfo, MultipartOperations,
MultipartUploadInspection, MultipartUploadResult, PartInfo,
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartAbortAuthorization, MultipartCompletionAuthorization,
MultipartInfo, MultipartOperations, MultipartOwner, MultipartUploadInspection, MultipartUploadResult, PartInfo,
};
}
+22
View File
@@ -6825,6 +6825,28 @@ mod tests {
tokio::fs::remove_file(conflicting_metadata)
.await
.expect("remove conflicting fixture");
struct RequiresOwner;
impl crate::storage_api_contracts::multipart::MultipartAbortAuthorization for RequiresOwner {
type Error = StorageError;
fn authorize(&self, _: Option<&crate::storage_api_contracts::multipart::MultipartOwner>) -> Result<()> {
panic!("metadata-absent recovery cannot observe an upload owner");
}
}
let native_opts = ObjectOptions {
multipart_abort_authorization: Some(Arc::new(RequiresOwner)),
..Default::default()
};
let rejected = store
.abort_multipart_upload(&bucket, object, &upload.upload_id, &native_opts)
.await;
for disk_index in 0..3 {
assert!(
staging_dir(0, disk_index).join("part.1").exists(),
"owner-required Abort must preserve late data"
);
}
assert!(matches!(rejected, Err(StorageError::InvalidUploadID(..))));
let (paused, release) = crate::store::multipart::pause_orphan_multipart_cleanup_for_test(&bucket);
let recovery_store = store.clone();
let recovery_bucket = bucket.clone();
+2 -1
View File
@@ -966,7 +966,8 @@ impl ECStore {
bucket_lifecycle_guard: Option<super::BucketIncarnationFenceGuard>,
) -> Result<()> {
let invalid_upload = || StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned());
if self.ctx.lock_manager().is_disabled() {
// Without upload metadata this path has no recorded owner to authorize.
if opts.multipart_abort_authorization.is_some() || self.ctx.lock_manager().is_disabled() {
return Err(invalid_upload());
}
let Some(bucket_lifecycle_guard) = bucket_lifecycle_guard else {
@@ -454,6 +454,7 @@ const INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL: &str = "rustfs_system_network_in
const INTERNODE_MSGPACK_JSON_DECODE_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_decode_total";
const INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_fallback_total";
const INTERNODE_MSGPACK_JSON_DECODE_ERROR_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_decode_error_total";
const INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL: &str = "rustfs_system_network_internode_msgpack_send_decisions_total";
const INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_signature_v1_fallback_total";
const INTERNODE_BODY_DIGEST_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_body_digest_fallback_total";
@@ -1604,6 +1605,25 @@ impl InternodeMetrics {
.increment(1);
}
/// Record one eligible request's fixed-cardinality JSON/msgpack send decision.
pub fn record_msgpack_send_decision(&self, decision: &'static str, reason: &'static str) {
let decision = match decision {
"msgpack_only" | "dual_write" => decision,
_ => "dual_write",
};
let reason = match reason {
"manual_fleet" | "peer_capable" | "peer_unknown" | "peer_revoked" | "not_requested" => reason,
_ => "peer_unknown",
};
counter!(
INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL,
SERVER_LABEL => current_server_label(),
"decision" => decision,
REASON_LABEL => reason
)
.increment(1);
}
#[doc(hidden)]
pub fn msgpack_json_decode_error_total_for_test(&self) -> u64 {
self.msgpack_json_decode_error_total.load(Ordering::Relaxed)
@@ -2700,6 +2720,52 @@ mod tests {
assert!(observed.contains(&(INTERNODE_MSGPACK_DIRECTION_RESPONSE.to_string(), "RawFileInfo".to_string(), 1)));
}
#[test]
fn msgpack_send_decision_counter_keeps_fixed_decision_and_reason_labels() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let metrics = InternodeMetrics::default();
metrics::with_local_recorder(&recorder, || {
for (decision, reason) in [
("msgpack_only", "manual_fleet"),
("msgpack_only", "peer_capable"),
("dual_write", "peer_unknown"),
("dual_write", "peer_revoked"),
("dual_write", "not_requested"),
] {
metrics.record_msgpack_send_decision(decision, reason);
}
metrics.record_msgpack_send_decision("untrusted", "untrusted");
});
let observed: HashMap<(String, String), u64> = snapshotter
.snapshot()
.into_vec()
.into_iter()
.filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL)
.map(|(composite, _, _, value)| {
let labels: HashMap<String, String> = composite
.key()
.labels()
.map(|label| (label.key().to_string(), label.value().to_string()))
.collect();
let count = match value {
DebugValue::Counter(count) => count,
other => panic!("msgpack send decision must be a counter, got {other:?}"),
};
(
(
labels.get("decision").cloned().unwrap_or_default(),
labels.get(REASON_LABEL).cloned().unwrap_or_default(),
),
count,
)
})
.collect();
assert_eq!(observed.len(), 5);
assert_eq!(observed.get(&("dual_write".to_string(), "peer_unknown".to_string())), Some(&2));
}
#[test]
fn msgpack_json_decode_counter_tracks_codec_direction_and_message() {
let metrics = InternodeMetrics::default();
+13 -2
View File
@@ -433,6 +433,7 @@ pub const PUT_COMMIT_LOCK_ADMISSION_OUTCOME_LOCK_ERROR: &str = "lock_error";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL: &str = "serial";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL: &str = "parallel";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_GROUPED: &str = "grouped";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER: &str = "leader";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER: &str = "follower";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT: &str = "follower_timeout";
@@ -4430,6 +4431,7 @@ mod tests {
set_put_stage_metrics_enabled(true);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL, 9);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_GROUPED, 4);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER, 1.0);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT, 3.0);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER, 2.0);
@@ -4445,7 +4447,10 @@ mod tests {
});
let rows = snapshotter.snapshot().into_vec();
assert_eq!(histogram_samples(&rows, "rustfs_s3_put_object_rename_fdatasync_batch_files"), vec![9.0]);
assert_eq!(
histogram_samples(&rows, "rustfs_s3_put_object_rename_fdatasync_batch_files"),
vec![4.0, 9.0]
);
let batch_modes = rows
.iter()
.filter(|(composite, _, _, _)| {
@@ -4461,7 +4466,13 @@ mod tests {
.collect::<Vec<_>>()
})
.collect::<HashSet<_>>();
assert_eq!(batch_modes, HashSet::from([PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL.to_string()]));
assert_eq!(
batch_modes,
HashSet::from([
PUT_RENAME_FDATASYNC_BATCH_MODE_GROUPED.to_string(),
PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL.to_string(),
])
);
let quorum_samples = histogram_samples(&rows, "rustfs_s3_put_object_rename_quorum_wait_fanout_disks");
assert_eq!(quorum_samples, vec![0.0, 1.0, 3.0, 3.0, 4.0]);
+7 -7
View File
@@ -73,7 +73,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "batch_read_version_req",
bin_field: "batch_read_version_req_bin",
},
json_encoder: "let batch_read_version_req = compat_json(&req)?;",
json_encoder: "let batch_read_version_req = compat_json(batch_rpc_req.as_ref(), &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -118,7 +118,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "read_multiple_req",
bin_field: "read_multiple_req_bin",
},
json_encoder: "let read_multiple_req = compat_json(&req)?;",
json_encoder: "let read_multiple_req = compat_json(&req, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -127,7 +127,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "opts",
bin_field: "opts_bin",
},
json_encoder: "let encoded_opts = compat_json(opts).and_then(|opts_str| encode_msgpack(opts).map(|opts_bin| (opts_str, opts_bin)));",
json_encoder: "let opts_str = compat_json(opts, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -136,7 +136,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi)?;",
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -145,7 +145,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi)?;",
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -154,7 +154,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "opts",
bin_field: "opts_bin",
},
json_encoder: "let opts_str = compat_json(&opts)?;",
json_encoder: "let opts_str = compat_json(opts, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -163,7 +163,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi)?;",
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
];
+54
View File
@@ -141,6 +141,7 @@ impl Drop for DialLock {
}
static INTERNODE_RPC_MSGPACK_ONLY_CACHE: AtomicU8 = AtomicU8::new(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET);
static INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE: AtomicU8 = AtomicU8::new(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET);
fn enforce_tls_generation_cache_bound(generation_cache: &mut HashMap<String, u64>, generation: u64, addr: &str) {
if generation_cache.len() < TLS_GENERATION_CACHE_MAX_SIZE || generation_cache.contains_key(addr) {
@@ -3422,9 +3423,35 @@ pub fn internode_rpc_msgpack_only() -> bool {
enabled
}
/// Whether msgpack-only was requested and the default-off per-peer capability gate is enabled.
pub fn internode_rpc_msgpack_peer_capability_enabled() -> bool {
match INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.load(Ordering::Acquire) {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_FALSE => return false,
INTERNODE_RPC_MSGPACK_ONLY_CACHE_TRUE => return true,
_ => {}
}
let enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_MSGPACK_ONLY,
rustfs_config::DEFAULT_INTERNODE_RPC_MSGPACK_ONLY,
) && rustfs_utils::get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY,
rustfs_config::DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY,
);
INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.store(
if enabled {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_TRUE
} else {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_FALSE
},
Ordering::Release,
);
enabled
}
#[doc(hidden)]
pub fn reset_internode_rpc_msgpack_only_cache() {
INTERNODE_RPC_MSGPACK_ONLY_CACHE.store(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET, Ordering::Release);
INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.store(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET, Ordering::Release);
}
/// Consecutive-failure threshold after which an internode peer is marked offline (grpc-optimization
@@ -5757,6 +5784,33 @@ mod tests {
reset_internode_rpc_msgpack_only_cache();
}
#[test]
fn internode_rpc_msgpack_peer_capability_requires_request_and_peer_gate() {
let _guard = INTERNODE_RPC_MSGPACK_ONLY_ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner());
for (requested, peer_capability, expected) in [
(None, None, false),
(Some("true"), None, false),
(None, Some("true"), false),
(Some("true"), Some("false"), false),
(Some("false"), Some("true"), false),
(Some("true"), Some("true"), true),
] {
reset_internode_rpc_msgpack_only_cache();
temp_env::with_vars(
[
(rustfs_config::ENV_INTERNODE_RPC_MSGPACK_ONLY, requested),
(rustfs_config::ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY, peer_capability),
],
|| {
assert_eq!(internode_rpc_msgpack_peer_capability_enabled(), expected);
},
);
}
reset_internode_rpc_msgpack_only_cache();
}
#[tokio::test]
async fn get_channel_for_class_bulk_reuses_control_cache_when_isolation_disabled() {
let _test_guard = TLS02_TEST_LOCK.lock().await;
+46 -1
View File
@@ -83,6 +83,13 @@ pub enum S3Operation {
SelectObjectContent,
UploadPart,
UploadPartCopy,
GetBucketAccelerateConfiguration,
GetBucketRequestPayment,
GetBucketWebsite,
PutBucketAccelerateConfiguration,
PutBucketRequestPayment,
PutBucketWebsite,
DeleteBucketWebsite,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -126,7 +133,7 @@ impl EventMapping {
}
impl S3Operation {
pub const ALL: [Self; 66] = [
pub const ALL: [Self; 73] = [
Self::AbortMultipartUpload,
Self::CompleteMultipartUpload,
Self::CopyObject,
@@ -193,6 +200,13 @@ impl S3Operation {
Self::SelectObjectContent,
Self::UploadPart,
Self::UploadPartCopy,
Self::GetBucketAccelerateConfiguration,
Self::GetBucketRequestPayment,
Self::GetBucketWebsite,
Self::PutBucketAccelerateConfiguration,
Self::PutBucketRequestPayment,
Self::PutBucketWebsite,
Self::DeleteBucketWebsite,
];
#[inline]
@@ -214,10 +228,12 @@ impl S3Operation {
Self::DeleteBucketPolicy => "s3:DeleteBucketPolicy",
Self::DeleteBucketReplication => "s3:DeleteBucketReplication",
Self::DeleteBucketTagging => "s3:DeleteBucketTagging",
Self::DeleteBucketWebsite => "s3:DeleteBucketWebsite",
Self::DeleteObject => "s3:DeleteObject",
Self::DeleteObjectTagging => "s3:DeleteObjectTagging",
Self::DeleteObjects => "s3:DeleteObjects",
Self::DeletePublicAccessBlock => "s3:DeletePublicAccessBlock",
Self::GetBucketAccelerateConfiguration => "s3:GetBucketAccelerateConfiguration",
Self::GetBucketAcl => "s3:GetBucketAcl",
Self::GetBucketCors => "s3:GetBucketCors",
Self::GetBucketEncryption => "s3:GetBucketEncryption",
@@ -228,8 +244,10 @@ impl S3Operation {
Self::GetBucketPolicy => "s3:GetBucketPolicy",
Self::GetBucketPolicyStatus => "s3:GetBucketPolicyStatus",
Self::GetBucketReplication => "s3:GetBucketReplication",
Self::GetBucketRequestPayment => "s3:GetBucketRequestPayment",
Self::GetBucketTagging => "s3:GetBucketTagging",
Self::GetBucketVersioning => "s3:GetBucketVersioning",
Self::GetBucketWebsite => "s3:GetBucketWebsite",
Self::GetObject => "s3:GetObject",
Self::GetObjectAcl => "s3:GetObjectAcl",
Self::GetObjectAttributes => "s3:GetObjectAttributes",
@@ -247,6 +265,7 @@ impl S3Operation {
Self::ListObjects => "s3:ListObjects",
Self::ListObjectsV2 => "s3:ListObjectsV2",
Self::ListParts => "s3:ListParts",
Self::PutBucketAccelerateConfiguration => "s3:PutBucketAccelerateConfiguration",
Self::PutBucketAcl => "s3:PutBucketAcl",
Self::PutBucketCors => "s3:PutBucketCors",
Self::PutBucketEncryption => "s3:PutBucketEncryption",
@@ -255,8 +274,10 @@ impl S3Operation {
Self::PutBucketNotificationConfiguration => "s3:PutBucketNotificationConfiguration",
Self::PutBucketPolicy => "s3:PutBucketPolicy",
Self::PutBucketReplication => "s3:PutBucketReplication",
Self::PutBucketRequestPayment => "s3:PutBucketRequestPayment",
Self::PutBucketTagging => "s3:PutBucketTagging",
Self::PutBucketVersioning => "s3:PutBucketVersioning",
Self::PutBucketWebsite => "s3:PutBucketWebsite",
Self::PutObject => "s3:PutObject",
Self::PutObjectAcl => "s3:PutObjectAcl",
Self::PutObjectLegalHold => "s3:PutObjectLegalHold",
@@ -301,7 +322,9 @@ impl S3Operation {
| Self::DeleteBucketPolicy
| Self::DeleteBucketReplication
| Self::DeleteBucketTagging
| Self::DeleteBucketWebsite
| Self::DeletePublicAccessBlock
| Self::GetBucketAccelerateConfiguration
| Self::GetBucketAcl
| Self::GetBucketCors
| Self::GetBucketEncryption
@@ -312,8 +335,10 @@ impl S3Operation {
| Self::GetBucketPolicy
| Self::GetBucketPolicyStatus
| Self::GetBucketReplication
| Self::GetBucketRequestPayment
| Self::GetBucketTagging
| Self::GetBucketVersioning
| Self::GetBucketWebsite
| Self::GetObjectAcl
| Self::GetObjectLockConfiguration
| Self::GetObjectTagging
@@ -326,6 +351,7 @@ impl S3Operation {
| Self::ListObjects
| Self::ListObjectsV2
| Self::ListParts
| Self::PutBucketAccelerateConfiguration
| Self::PutBucketAcl
| Self::PutBucketCors
| Self::PutBucketEncryption
@@ -334,8 +360,10 @@ impl S3Operation {
| Self::PutBucketNotificationConfiguration
| Self::PutBucketPolicy
| Self::PutBucketReplication
| Self::PutBucketRequestPayment
| Self::PutBucketTagging
| Self::PutBucketVersioning
| Self::PutBucketWebsite
| Self::PutObjectLockConfiguration
| Self::PutPublicAccessBlock
| Self::UploadPart
@@ -516,12 +544,29 @@ mod tests {
assert_eq!(mask & EventName::ObjectRemovedDelete.mask(), 0);
}
#[test]
fn metric_indexes_and_labels_identify_each_operation() {
let mut labels = std::collections::HashSet::new();
for (index, op) in S3Operation::ALL.into_iter().enumerate() {
assert_eq!(op.metric_index(), index, "metric index for {op:?}");
assert!(labels.insert(op.as_str()), "repeated metric label for {op:?}");
}
assert_eq!(S3Operation::UploadPartCopy.metric_index(), 65);
}
#[test]
fn test_operations_without_event_mapping_remain_unmapped() {
let unmapped = [
S3Operation::DeleteBucketWebsite,
S3Operation::GetBucketAccelerateConfiguration,
S3Operation::GetBucketAcl,
S3Operation::GetBucketRequestPayment,
S3Operation::GetBucketWebsite,
S3Operation::ListObjectsV2,
S3Operation::PutBucketAccelerateConfiguration,
S3Operation::PutBucketNotificationConfiguration,
S3Operation::PutBucketRequestPayment,
S3Operation::PutBucketWebsite,
S3Operation::UploadPart,
];
for op in unmapped {
-1
View File
@@ -15,7 +15,6 @@
pub mod init_time;
pub mod last_minute;
pub mod metrics;
pub mod scan_observation;
pub mod startup;
pub mod usage_candidate;
+1 -40
View File
@@ -13,9 +13,6 @@
// limitations under the License.
use crate::last_minute::{AccElem, LastMinuteLatency};
use crate::scan_observation::{
ScannerMetadataReadGuard, ScannerMetadataReadOutcome, ScannerScanObservation, ScannerScanObservationSnapshot,
};
use crate::usage_candidate::{
ScannerUsageCandidateCount, ScannerUsageCandidateKind, ScannerUsageCandidateOutcome, USAGE_CANDIDATE_COUNTER_SLOTS,
};
@@ -1002,7 +999,6 @@ impl ScannerUsagePublicationObservation {
pub struct Metrics {
startup: crate::startup::ScannerStartupObservation,
scan_observation: ScannerScanObservation,
operations: Vec<AtomicU64>,
latency: Vec<LockedLastMinuteLatency>,
actions: Vec<AtomicU64>,
@@ -1657,9 +1653,6 @@ pub struct ScannerMetricsReport {
pub lifecycle_transition: ScannerLifecycleTransitionSnapshot,
#[serde(default)]
pub usage_freshness: ScannerUsageFreshnessSnapshot,
/// Node-local, process-lifetime work observations. Older nodes have no evidence.
#[serde(default)]
pub scan_observation: ScannerScanObservationSnapshot,
/// Process-lifetime candidate-construction results. Missing on older reports
/// means unobserved, not accepted or healthy. Publication is tracked separately.
#[serde(default)]
@@ -2154,38 +2147,9 @@ impl Metrics {
&self.startup
}
pub fn scan_observation(&self) -> &ScannerScanObservation {
&self.scan_observation
}
/// Start after scanner node admission, before polling the disk read future.
/// Disk-permit and RPC waits remain included; decoding is excluded. Dropping
/// an unfinished guard observes cancellation, not physical I/O completion.
pub fn start_scanner_metadata_read(&self) -> ScannerMetadataReadGuard<'_> {
self.scan_observation.start_metadata_read();
ScannerMetadataReadGuard {
metrics: self,
started: std::time::Instant::now(),
outcome: ScannerMetadataReadOutcome::Cancelled,
bytes: 0,
}
}
pub(crate) fn finish_scanner_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
let bytes = if outcome == ScannerMetadataReadOutcome::Success {
bytes
} else {
0
};
self.scan_observation.finish_metadata_read(outcome, bytes, elapsed);
self.operations[Metric::ReadMetadata as usize].fetch_add(1, Ordering::Relaxed);
self.latency[Metric::ReadMetadata as usize].add_size(elapsed, bytes);
}
pub fn new() -> Self {
Self {
startup: crate::startup::ScannerStartupObservation::default(),
scan_observation: ScannerScanObservation::default(),
operations: (0..Metric::Last as usize).map(|_| AtomicU64::new(0)).collect(),
// Each slot gets its own fresh LockedLastMinuteLatency so that
// different metrics never accidentally share state.
@@ -2791,8 +2755,6 @@ impl Metrics {
/// walked the whole namespace, so an operator can tell why scoped
/// convergence is not happening.
pub fn record_scanner_scope_fallback(&self, reason: impl Into<String>) {
let reason = reason.into();
self.scan_observation.record_scope_fallback(&reason);
self.scanner_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
self.scanner_scope_fallback_last_unix_secs
.store(unix_now_secs(), Ordering::Relaxed);
@@ -2800,7 +2762,7 @@ impl Metrics {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
*last_reason = reason;
*last_reason = reason.into();
}
pub fn record_scanner_usage_durable_success(&self) {
@@ -3935,7 +3897,6 @@ impl Metrics {
failed: self.scanner_transition_failed.load(Ordering::Relaxed),
};
m.usage_candidates = self.usage_candidate_counts();
m.scan_observation = self.scan_observation.snapshot();
let usage_save_result = ScannerUsageSaveResult::from_code(self.scanner_usage_last_save_result.load(Ordering::Relaxed));
m.usage_freshness = ScannerUsageFreshnessSnapshot {
dirty_pending_buckets: self.scanner_dirty_usage_pending_buckets.load(Ordering::Relaxed),
@@ -1,675 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Node-local, process-lifetime observations of scanner work. Counters are
//! independently loaded, so concurrent reports need not balance exactly.
//! These observations never grant cache reuse or publication authority.
use crate::metrics::Metrics;
use serde::{Deserialize, Serialize};
use std::{
collections::BTreeMap,
sync::atomic::{AtomicU64, Ordering},
time::{Duration, Instant},
};
pub const PREFIX_REUSE_BLOCKERS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
const SCOPE_FALLBACK_REASONS: [&str; 6] = [
"dirty_snapshot_incomplete",
"dirty_generation_exhausted",
"dirty_bucket_overflow",
"no_listed_dirty_buckets",
"baseline_unavailable",
"peer_scope_unavailable",
];
/// A storage read result, independent of later metadata decoding or ILM work.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerMetadataReadOutcome {
Success,
NotFound,
Error,
Cancelled,
}
/// The existing path that actually copied a cached subtree instead of walking it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseKind {
CleanPrefix,
CoveredNamespace,
SampledNamespace,
ClassifiedNamespace,
}
impl ScannerCacheReuseKind {
const ALL: [Self; 4] = [
Self::CleanPrefix,
Self::CoveredNamespace,
Self::SampledNamespace,
Self::ClassifiedNamespace,
];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::SampledNamespace => "sampled_namespace",
Self::ClassifiedNamespace => "classified_namespace",
}
}
}
/// Completed bucket-level reuse work, independent of durable publication.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBucketReuseKind {
ScopedColdBucket,
CurrentRoot,
}
impl ScannerBucketReuseKind {
const ALL: [Self; 2] = [Self::ScopedColdBucket, Self::CurrentRoot];
fn label(self) -> &'static str {
match self {
Self::ScopedColdBucket => "scoped_cold_bucket",
Self::CurrentRoot => "current_root",
}
}
}
/// A production folder gate, not a diagnostic preflight.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseGate {
CleanPrefix,
CoveredNamespace,
NamespaceSampling,
}
impl ScannerCacheReuseGate {
const ALL: [Self; 3] = [Self::CleanPrefix, Self::CoveredNamespace, Self::NamespaceSampling];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::NamespaceSampling => "namespace_sampling",
}
}
}
/// The first existing condition reached by a folder reuse gate.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseDecision {
MissingScope,
NoResumeSweep,
DeepScan,
ForwardSweep,
ActiveReplication,
NotRoot,
CompactedParent,
MissingCachedChild,
InvalidPrefix,
DirtyPrefix,
ErasureHealthRequired,
MissingCoverage,
CachedFailures,
UncompactedChild,
SelectedForWalk,
Eligible,
}
impl ScannerCacheReuseDecision {
const ALL: [Self; 16] = [
Self::MissingScope,
Self::NoResumeSweep,
Self::DeepScan,
Self::ForwardSweep,
Self::ActiveReplication,
Self::NotRoot,
Self::CompactedParent,
Self::MissingCachedChild,
Self::InvalidPrefix,
Self::DirtyPrefix,
Self::ErasureHealthRequired,
Self::MissingCoverage,
Self::CachedFailures,
Self::UncompactedChild,
Self::SelectedForWalk,
Self::Eligible,
];
fn label(self) -> &'static str {
match self {
Self::MissingScope => "missing_scope",
Self::NoResumeSweep => "no_resume_sweep",
Self::DeepScan => "deep_scan",
Self::ForwardSweep => "forward_sweep",
Self::ActiveReplication => "active_replication",
Self::NotRoot => "not_root",
Self::CompactedParent => "compacted_parent",
Self::MissingCachedChild => "missing_cached_child",
Self::InvalidPrefix => "invalid_prefix",
Self::DirtyPrefix => "dirty_prefix",
Self::ErasureHealthRequired => "erasure_health_required",
Self::MissingCoverage => "missing_coverage",
Self::CachedFailures => "cached_failures",
Self::UncompactedChild => "uncompacted_child",
Self::SelectedForWalk => "selected_for_walk",
Self::Eligible => "eligible",
}
}
fn not_evaluated(self) -> bool {
matches!(
self,
Self::MissingScope | Self::NoResumeSweep | Self::DeepScan | Self::ForwardSweep | Self::ActiveReplication
)
}
}
/// Scope admission and end-of-cycle evidence are separate observations. An
/// activated completion proof does not mean this cycle used prefix reuse.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerPrefixReuseStage {
ScopeResolution,
CycleCompletion,
}
/// The first existing baseline validation check that rejected a snapshot.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBaselineRejection {
Missing,
Decode,
Incomplete,
Partial,
Marks,
Epoch,
SourceCount,
PlanMissing,
PlanMismatch,
SourceIndex,
SourceMembership,
SourceDuplicate,
SetIncomplete,
SetTombstone,
SetEpoch,
SetCycle,
SetPlan,
SourceCoverage,
AuthoritativeIdentity,
NotNewer,
}
impl ScannerBaselineRejection {
const ALL: [Self; 20] = [
Self::Missing,
Self::Decode,
Self::Incomplete,
Self::Partial,
Self::Marks,
Self::Epoch,
Self::SourceCount,
Self::PlanMissing,
Self::PlanMismatch,
Self::SourceIndex,
Self::SourceMembership,
Self::SourceDuplicate,
Self::SetIncomplete,
Self::SetTombstone,
Self::SetEpoch,
Self::SetCycle,
Self::SetPlan,
Self::SourceCoverage,
Self::AuthoritativeIdentity,
Self::NotNewer,
];
fn label(self) -> &'static str {
match self {
Self::Missing => "missing",
Self::Decode => "decode",
Self::Incomplete => "incomplete",
Self::Partial => "partial",
Self::Marks => "marks",
Self::Epoch => "epoch",
Self::SourceCount => "source_count",
Self::PlanMissing => "plan_missing",
Self::PlanMismatch => "plan_mismatch",
Self::SourceIndex => "source_index",
Self::SourceMembership => "source_membership",
Self::SourceDuplicate => "source_duplicate",
Self::SetIncomplete => "set_incomplete",
Self::SetTombstone => "set_tombstone",
Self::SetEpoch => "set_epoch",
Self::SetCycle => "set_cycle",
Self::SetPlan => "set_plan",
Self::SourceCoverage => "source_coverage",
Self::AuthoritativeIdentity => "authoritative_identity",
Self::NotNewer => "not_newer",
}
}
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerMetadataReadSnapshot {
pub attempts: u64,
pub succeeded: u64,
pub not_found: u64,
pub errors: u64,
pub cancelled: u64,
/// Bytes returned by successful storage reads, including corrupt metadata.
pub bytes: u64,
/// Elapsed read-future time, including failed and cancelled reads.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseSnapshot {
/// Completed subtree reuse decisions; not cache entries or objects skipped.
pub count: u64,
/// Time spent copying and attaching the reused subtree, excluding listing.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerBucketReuseSnapshot {
/// Cold bucket copies or current roots successfully handed to the collector.
pub count: u64,
/// Cold copy/attachment time or current-root channel-send time; excludes validation.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseDecisionSnapshot {
/// Policy prerequisites that stopped the gate before cache proof checks.
pub not_evaluated: BTreeMap<String, u64>,
/// First failed cache/work condition; later conditions are not evaluated.
pub blocked: BTreeMap<String, u64>,
/// Admitted candidates, which may still need EC classification before copying.
pub eligible: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerPrefixReuseSnapshot {
pub evaluated: u64,
pub activated: u64,
/// All failed checks are counted; one evaluation can have several blockers.
pub blockers: BTreeMap<String, u64>,
}
/// Missing on an older node means unobserved, not zero work or healthy reuse.
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerScanObservationSnapshot {
pub metadata_reads: ScannerMetadataReadSnapshot,
pub cache_reuse: BTreeMap<String, ScannerCacheReuseSnapshot>,
pub bucket_reuse: BTreeMap<String, ScannerBucketReuseSnapshot>,
pub reuse_decisions: BTreeMap<String, ScannerCacheReuseDecisionSnapshot>,
/// Clean root children still walked because EC scans also check health.
pub erasure_clean_prefix_walks: u64,
pub prefix_scope_resolution: ScannerPrefixReuseSnapshot,
pub prefix_cycle_completion: ScannerPrefixReuseSnapshot,
pub scope_fallbacks: BTreeMap<String, u64>,
/// Rejections only for cycles with a baseline_unavailable scope fallback.
pub authoritative_baseline_rejections: BTreeMap<String, u64>,
pub observed_baseline_rejections: BTreeMap<String, u64>,
}
#[derive(Default)]
pub struct ScannerScanObservation {
metadata: [AtomicU64; 7],
cache_count: [AtomicU64; 4],
cache_nanos: [AtomicU64; 4],
bucket_count: [AtomicU64; 2],
bucket_nanos: [AtomicU64; 2],
reuse_decisions: [[AtomicU64; ScannerCacheReuseDecision::ALL.len()]; 3],
erasure_clean_prefix_walks: AtomicU64,
prefix_evaluated: [AtomicU64; 2],
prefix_activated: [AtomicU64; 2],
prefix_blockers: [[AtomicU64; PREFIX_REUSE_BLOCKERS.len()]; 2],
scope_fallbacks: [AtomicU64; SCOPE_FALLBACK_REASONS.len()],
baseline_rejections: [[AtomicU64; ScannerBaselineRejection::ALL.len()]; 2],
}
fn duration_nanos(duration: Duration) -> u64 {
u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX)
}
fn labelled_counts(labels: impl IntoIterator<Item = &'static str>, counters: &[AtomicU64]) -> BTreeMap<String, u64> {
labels
.into_iter()
.zip(counters)
.map(|(label, count)| (label.to_owned(), count.load(Ordering::Relaxed)))
.collect()
}
impl ScannerScanObservation {
pub(crate) fn start_metadata_read(&self) {
self.metadata[0].fetch_add(1, Ordering::Relaxed);
}
pub(crate) fn finish_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
self.metadata[1 + outcome as usize].fetch_add(1, Ordering::Relaxed);
if outcome == ScannerMetadataReadOutcome::Success {
self.metadata[5].fetch_add(bytes, Ordering::Relaxed);
}
self.metadata[6].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
pub fn record_cache_reuse(&self, kind: ScannerCacheReuseKind, elapsed: Duration) {
self.cache_count[kind as usize].fetch_add(1, Ordering::Relaxed);
self.cache_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records completed reuse work only; it grants no publication authority.
pub fn record_bucket_reuse(&self, kind: ScannerBucketReuseKind, count: u64, elapsed: Duration) {
self.bucket_count[kind as usize].fetch_add(count, Ordering::Relaxed);
self.bucket_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records one first-condition decision without inspecting later prerequisites.
pub fn record_cache_reuse_decision(&self, gate: ScannerCacheReuseGate, decision: ScannerCacheReuseDecision) {
self.reuse_decisions[gate as usize][decision as usize].fetch_add(1, Ordering::Relaxed);
}
pub fn record_erasure_clean_prefix_walk(&self) {
self.erasure_clean_prefix_walks.fetch_add(1, Ordering::Relaxed);
}
pub fn record_prefix_reuse(&self, stage: ScannerPrefixReuseStage, activated: bool, blockers: &[Option<&str>; 8]) {
let stage = stage as usize;
self.prefix_evaluated[stage].fetch_add(1, Ordering::Relaxed);
if activated {
self.prefix_activated[stage].fetch_add(1, Ordering::Relaxed);
}
for blocker in blockers.iter().flatten() {
if let Some(index) = PREFIX_REUSE_BLOCKERS.iter().position(|label| label == blocker) {
self.prefix_blockers[stage][index].fetch_add(1, Ordering::Relaxed);
}
}
}
pub(crate) fn record_scope_fallback(&self, reason: &str) {
if let Some(index) = SCOPE_FALLBACK_REASONS.iter().position(|label| *label == reason) {
self.scope_fallbacks[index].fetch_add(1, Ordering::Relaxed);
}
}
pub fn record_baseline_rejection(&self, authoritative: ScannerBaselineRejection, observed: Option<ScannerBaselineRejection>) {
self.baseline_rejections[0][authoritative as usize].fetch_add(1, Ordering::Relaxed);
if let Some(observed) = observed {
self.baseline_rejections[1][observed as usize].fetch_add(1, Ordering::Relaxed);
}
}
pub fn snapshot(&self) -> ScannerScanObservationSnapshot {
let prefix = |stage: ScannerPrefixReuseStage| ScannerPrefixReuseSnapshot {
evaluated: self.prefix_evaluated[stage as usize].load(Ordering::Relaxed),
activated: self.prefix_activated[stage as usize].load(Ordering::Relaxed),
blockers: labelled_counts(PREFIX_REUSE_BLOCKERS, &self.prefix_blockers[stage as usize]),
};
ScannerScanObservationSnapshot {
metadata_reads: ScannerMetadataReadSnapshot {
attempts: self.metadata[0].load(Ordering::Relaxed),
succeeded: self.metadata[1].load(Ordering::Relaxed),
not_found: self.metadata[2].load(Ordering::Relaxed),
errors: self.metadata[3].load(Ordering::Relaxed),
cancelled: self.metadata[4].load(Ordering::Relaxed),
bytes: self.metadata[5].load(Ordering::Relaxed),
elapsed_nanos: self.metadata[6].load(Ordering::Relaxed),
},
cache_reuse: ScannerCacheReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerCacheReuseSnapshot {
count: self.cache_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.cache_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
bucket_reuse: ScannerBucketReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerBucketReuseSnapshot {
count: self.bucket_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.bucket_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
reuse_decisions: ScannerCacheReuseGate::ALL
.into_iter()
.map(|gate| {
let counts = &self.reuse_decisions[gate as usize];
let decisions = |not_evaluated| {
ScannerCacheReuseDecision::ALL
.into_iter()
.filter(|decision| {
*decision != ScannerCacheReuseDecision::Eligible && decision.not_evaluated() == not_evaluated
})
.map(|decision| (decision.label().to_owned(), counts[decision as usize].load(Ordering::Relaxed)))
.collect()
};
(
gate.label().to_owned(),
ScannerCacheReuseDecisionSnapshot {
not_evaluated: decisions(true),
blocked: decisions(false),
eligible: counts[ScannerCacheReuseDecision::Eligible as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
erasure_clean_prefix_walks: self.erasure_clean_prefix_walks.load(Ordering::Relaxed),
prefix_scope_resolution: prefix(ScannerPrefixReuseStage::ScopeResolution),
prefix_cycle_completion: prefix(ScannerPrefixReuseStage::CycleCompletion),
scope_fallbacks: labelled_counts(SCOPE_FALLBACK_REASONS, &self.scope_fallbacks),
authoritative_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[0],
),
observed_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[1],
),
}
}
}
/// Dropping an unfinished read records cancellation. The guard observes the
/// read future only; it cannot cancel it or change its storage result.
#[must_use]
pub struct ScannerMetadataReadGuard<'a> {
pub(crate) metrics: &'a Metrics,
pub(crate) started: Instant,
pub(crate) outcome: ScannerMetadataReadOutcome,
pub(crate) bytes: u64,
}
impl ScannerMetadataReadGuard<'_> {
pub fn finish(mut self, outcome: ScannerMetadataReadOutcome, bytes: u64) {
self.outcome = outcome;
self.bytes = bytes;
}
}
impl Drop for ScannerMetadataReadGuard<'_> {
fn drop(&mut self) {
self.metrics
.finish_scanner_metadata_read(self.outcome, self.bytes, self.started.elapsed());
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[tokio::test]
async fn metadata_read_reports_storage_results_and_existing_read_metric() {
let metrics = Metrics::new();
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 127);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::NotFound, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Error, 0);
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!(
(reads.attempts, reads.succeeded, reads.not_found, reads.errors, reads.cancelled),
(4, 2, 1, 1, 0)
);
assert_eq!(reads.bytes, 127);
assert_eq!(report.life_time_ops["read_metadata"], 4);
assert_eq!(report.last_minute.actions["read_metadata"].count, 4);
assert_eq!(report.last_minute.actions["read_metadata"].bytes, 127);
}
#[tokio::test]
async fn metadata_read_cancellation_balances_an_actually_polled_future_once() {
let metrics = Arc::new(Metrics::new());
let worker_metrics = Arc::clone(&metrics);
let (started, admitted) = tokio::sync::oneshot::channel();
let worker = tokio::spawn(async move {
let read = worker_metrics.start_scanner_metadata_read();
started.send(()).expect("observe started read");
std::future::pending::<()>().await;
read.finish(ScannerMetadataReadOutcome::Success, 99);
});
tokio::time::timeout(Duration::from_secs(5), admitted)
.await
.expect("read starts within deadline")
.expect("read started");
let in_flight = metrics.scan_observation().snapshot().metadata_reads;
assert_eq!((in_flight.attempts, in_flight.cancelled, in_flight.succeeded), (1, 0, 0));
worker.abort();
assert!(worker.await.expect_err("read task was aborted").is_cancelled());
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!((reads.attempts, reads.cancelled, reads.succeeded, reads.bytes), (1, 1, 0, 0));
assert_eq!(report.life_time_ops["read_metadata"], 1);
assert_eq!(report.last_minute.actions["read_metadata"].count, 1);
assert_eq!(
Duration::from_nanos(reads.elapsed_nanos).as_secs(),
report.last_minute.actions["read_metadata"].acc_time
);
}
#[tokio::test]
async fn observation_report_keeps_stages_reasons_and_copy_costs_separate() {
let metrics = Metrics::new();
let observation = metrics.scan_observation();
observation.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, Duration::from_nanos(321));
observation.record_bucket_reuse(ScannerBucketReuseKind::ScopedColdBucket, 2, Duration::from_nanos(654));
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, Duration::from_nanos(987));
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::MissingScope);
observation
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::ErasureHealthRequired);
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, ScannerCacheReuseDecision::Eligible);
observation.record_erasure_clean_prefix_walk();
observation.record_prefix_reuse(
ScannerPrefixReuseStage::ScopeResolution,
false,
&[
Some("missing_producer_identity"),
Some("missing_durable_journal_replay"),
None,
None,
None,
None,
None,
None,
],
);
observation.record_prefix_reuse(ScannerPrefixReuseStage::CycleCompletion, true, &[None; 8]);
metrics.record_scanner_scope_fallback("baseline_unavailable");
observation.record_baseline_rejection(ScannerBaselineRejection::Epoch, Some(ScannerBaselineRejection::Missing));
let report = metrics.report().await;
let snapshot = &report.scan_observation;
assert_eq!(snapshot.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.cache_reuse["clean_prefix"].elapsed_nanos, 321);
assert_eq!(snapshot.cache_reuse["sampled_namespace"].count, 0);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].count, 2);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].elapsed_nanos, 654);
assert_eq!(snapshot.bucket_reuse["current_root"].count, 1);
assert_eq!(snapshot.bucket_reuse["current_root"].elapsed_nanos, 987);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"], 1);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].blocked["erasure_health_required"], 1);
assert_eq!(snapshot.reuse_decisions["covered_namespace"].eligible, 1);
let mut prior_observations = serde_json::to_value(snapshot).expect("encode observations");
let fields = prior_observations.as_object_mut().expect("observation object");
fields.remove("bucket_reuse");
fields.remove("reuse_decisions");
let prior: ScannerScanObservationSnapshot =
serde_json::from_value(prior_observations).expect("read earlier observations");
assert!(prior.bucket_reuse.is_empty());
assert!(prior.reuse_decisions.is_empty());
assert_eq!(prior.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.erasure_clean_prefix_walks, 1);
assert_eq!(
(snapshot.prefix_scope_resolution.evaluated, snapshot.prefix_scope_resolution.activated),
(1, 0)
);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_producer_identity"], 1);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_durable_journal_replay"], 1);
assert_eq!(
(snapshot.prefix_cycle_completion.evaluated, snapshot.prefix_cycle_completion.activated),
(1, 1)
);
assert_eq!(
snapshot.scope_fallbacks["baseline_unavailable"],
report.usage_freshness.scope_fallback_total
);
assert_eq!(snapshot.authoritative_baseline_rejections["epoch"], 1);
assert_eq!(snapshot.observed_baseline_rejections["missing"], 1);
let mut legacy = serde_json::to_value(&report).expect("encode report");
legacy
.as_object_mut()
.expect("report is an object")
.remove("scan_observation");
let decoded: crate::metrics::ScannerMetricsReport = serde_json::from_value(legacy).expect("read legacy report");
assert!(decoded.scan_observation.cache_reuse.is_empty(), "legacy counters are unobserved");
assert_eq!(decoded.scan_observation.metadata_reads.attempts, 0);
}
}
-26
View File
@@ -206,32 +206,6 @@ pub(crate) struct DataUsageCacheRevisions {
backup: Option<DataUsageCacheRevision>,
}
/// In-memory provenance of the decoded cache and the revisions observed by its loader.
#[derive(Clone, Debug)]
pub(crate) struct DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions,
loaded_from_main: bool,
}
impl DataUsageCacheBaselineRevisions {
pub(crate) fn revisions(&self) -> &DataUsageCacheRevisions {
&self.revisions
}
/// A backup may catch up to an unchanged primary without changing the decoded baseline.
/// Both reads must have decoded that primary; an ETag on a corrupt primary is insufficient.
pub(crate) fn permits_unchanged_baseline_since(&self, initial: &Self) -> bool {
if self.revisions == initial.revisions {
return true;
}
self.loaded_from_main
&& initial.loaded_from_main
&& matches!(&initial.revisions.main, DataUsageCacheRevision::Etag(etag) if !etag.is_empty())
&& self.revisions.main == initial.revisions.main
&& self.revisions.backup.as_ref() == Some(&self.revisions.main)
}
}
pub static DATA_USAGE_BUCKET: LazyLock<String> =
LazyLock::new(|| format!("{RUSTFS_META_BUCKET}{SLASH_SEPARATOR}{BUCKET_META_PREFIX}"));
@@ -70,7 +70,6 @@ struct DataUsageCacheLoadResult {
cache: DataUsageCache,
main_revision: DataUsageCacheRevision,
backup_revision: Option<DataUsageCacheRevision>,
loaded_from_main: bool,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -107,18 +106,6 @@ impl DataUsageCache {
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheRevisions> {
self.load_with_baseline_revisions(store, name)
.await
.map(|baseline| baseline.revisions)
}
/// Loads cache revisions with provenance for unchanged-primary backup reconciliation.
/// Missing or rebuilt caches and a decoded backup do not establish a decoded primary.
pub(crate) async fn load_with_baseline_revisions<S: ScannerObjectIO>(
&mut self,
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheBaselineRevisions> {
let backup_name = format!("{name}.bkp");
let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]);
let loaded = Self::load_cache(store.clone(), name).await?;
@@ -146,10 +133,7 @@ impl DataUsageCache {
let main = loaded.main_revision;
*self = loaded.cache;
Ok(DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions { main, backup },
loaded_from_main: loaded.loaded_from_main,
})
Ok(DataUsageCacheRevisions { main, backup })
}
pub(crate) async fn read_revisions<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheRevisions> {
@@ -172,13 +156,10 @@ impl DataUsageCache {
cache,
revision: Some(main_revision),
} => {
// Legacy-location fallback carries Missing for the primary path.
let loaded_from_main = matches!(&main_revision, DataUsageCacheRevision::Etag(_));
return Ok(DataUsageCacheLoadResult {
cache: *cache,
main_revision,
backup_revision: None,
loaded_from_main,
});
}
DataUsageCacheLoadAttempt::Loaded { revision: None, .. } => {
@@ -216,7 +197,6 @@ impl DataUsageCache {
cache: *cache,
main_revision,
backup_revision: Some(backup_revision),
loaded_from_main: false,
});
}
DataUsageCacheLoadState::Loaded
@@ -240,7 +220,6 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache missing state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
(DataUsageCacheLoadState::Corrupt, DataUsageCacheLoadState::Missing)
@@ -260,7 +239,6 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache corrupt state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
_ => {}
@@ -772,102 +772,6 @@ async fn corrupt_primary_cache_loads_valid_backup() {
assert_eq!(store.puts.load(Ordering::SeqCst), 0);
}
#[test]
fn baseline_revisions_allow_only_a_decoded_unchanged_primary_backup_catchup() {
let etag = |value: &str| DataUsageCacheRevision::Etag(value.to_owned());
let initial = DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions {
main: etag("primary"),
backup: Some(etag("old-backup")),
},
loaded_from_main: true,
};
assert!(initial.permits_unchanged_baseline_since(&initial));
let mut caught_up = initial.clone();
caught_up.revisions.backup = Some(etag("primary"));
assert!(caught_up.permits_unchanged_baseline_since(&initial));
assert!(
!initial.permits_unchanged_baseline_since(&caught_up),
"backup rollback changes the baseline"
);
for backup in [None, Some(DataUsageCacheRevision::Missing)] {
let mut missing_backup = initial.clone();
missing_backup.revisions.backup = backup;
assert!(caught_up.permits_unchanged_baseline_since(&missing_backup));
assert!(!missing_backup.permits_unchanged_baseline_since(&initial));
}
for (initial_from_main, current_from_main) in [(false, true), (true, false), (false, false)] {
let mut baseline = initial.clone();
baseline.loaded_from_main = initial_from_main;
let mut current = caught_up.clone();
current.loaded_from_main = current_from_main;
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"backup fallback grants no primary proof"
);
current.revisions = baseline.revisions.clone();
assert!(
current.permits_unchanged_baseline_since(&baseline),
"exact revisions retain existing admission"
);
}
let mut changed_main = caught_up.clone();
changed_main.revisions.main = etag("new-primary");
changed_main.revisions.backup = Some(etag("new-primary"));
assert!(!changed_main.permits_unchanged_baseline_since(&initial));
let mut changed_backup = caught_up;
changed_backup.revisions.backup = Some(etag("unrelated-backup"));
assert!(!changed_backup.permits_unchanged_baseline_since(&initial));
for main in [DataUsageCacheRevision::Missing, etag("")] {
let mut baseline = initial.clone();
baseline.revisions.main = main.clone();
let mut current = baseline.clone();
current.revisions.backup = Some(main);
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"missing or empty primary ETag is not proof"
);
}
}
#[tokio::test]
async fn baseline_revisions_record_the_actual_decoded_cache_source() {
let mut expected = DataUsageCache::default();
expected.info.name = "decoded".to_owned();
let valid = expected.marshal_msg().expect("encode valid cache");
for (main, backup, from_main, name) in [
(valid.clone(), Some(valid.clone()), true, "decoded"),
(b"corrupt".to_vec(), Some(valid.clone()), false, "decoded"),
(b"corrupt".to_vec(), None, false, ""),
(Vec::new(), None, false, ""),
] {
let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(main), backup));
let mut loaded = DataUsageCache::default();
let baseline = loaded
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
.expect("load actual cache source");
assert_eq!(baseline.loaded_from_main, from_main);
assert_eq!(loaded.info.name, name);
let mut raw_loaded = DataUsageCache::default();
let raw = raw_loaded
.load_with_revisions(store, DATA_USAGE_CACHE_NAME)
.await
.expect("load compatible raw revisions");
assert_eq!(baseline.revisions(), &raw, "raw loader retains its revision contract");
}
let mut legacy_store = CacheReadStore::new(CacheReadBody::Bytes(Vec::new()), None);
legacy_store.legacy = Some(valid);
let mut legacy_cache = DataUsageCache::default();
let legacy = legacy_cache
.load_with_baseline_revisions(Arc::new(legacy_store), "legacy-cache")
.await
.expect("load cache from the legacy location");
assert_eq!(legacy_cache.info.name, "decoded");
assert!(!legacy.loaded_from_main);
assert_eq!(legacy.revisions.main, DataUsageCacheRevision::Missing);
}
#[tokio::test]
async fn corrupt_cache_without_valid_backup_rebuilds_with_cas_revisions() {
for backup in [None, Some(b"also-not-msgpack".to_vec())] {
+9 -8
View File
@@ -248,13 +248,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2;
/// otherwise idle instance whose clean-idle backoff had doubled a 60 s
/// interval), which defeats the fast path it is meant to protect.
///
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] protects
/// against a persistently hot bucket driving an unbroken full-scan loop, so
/// retries can start small: 5 s, 10 s, 20 s … Superseded retries cap this
/// schedule at the ordinary cycle interval and preserve an operator-selected
/// nominal cadence. Preflight deferrals retain the generic
/// [`SCANNER_RETRY_MAX_INTERVAL`] cap so a temporarily unavailable peer cannot
/// drive a tight retry loop.
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] is
/// what protects against a persistently hot bucket driving an unbroken
/// full-scan loop, so it can start small: 5 s, 10 s, 20 s … capped by
/// [`SCANNER_RETRY_MAX_INTERVAL`]. A one-off race recovers in seconds; a
/// genuinely hot bucket still reaches minute-scale backoff within a handful of
/// cycles. Preflight deferrals use the same bounded schedule so a temporarily
/// unavailable peer cannot drive a tight retry loop.
const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5);
const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60);
/// Probe cadence while data movement pauses the scanner. Recovery retries use
@@ -3536,7 +3536,8 @@ where
));
let mut wait_plan =
scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for);
let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config);
let superseded_retry_interval =
scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features);
let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval);
let scope_fallback_retry_interval = scanner_scope_fallback_retry_interval(scope_fallback_backoff, &runtime_config);
let mut convergence_retry_interval = superseded_retry_interval
+10 -7
View File
@@ -192,16 +192,19 @@ impl ScannerRetryBackoff {
pub(super) fn scanner_superseded_retry_interval(
backoff: ScannerRetryBackoff,
runtime_config: &ScannerRuntimeConfig,
maintenance_features: ScannerMaintenanceFeatures,
) -> Option<Duration> {
let cycle_interval = runtime_config.cycle_interval.max(Duration::from_secs(1));
let retry_interval = backoff.retry_interval(cycle_interval)?;
// Preserve an operator-selected nominal cadence during convergence.
let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?;
// Preserve an operator-selected duty-cycle floor during convergence.
if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default {
return Some(cycle_interval);
retry_interval = retry_interval.max(runtime_config.cycle_interval);
}
// A complete but non-authoritative walk must not postpone convergence
// beyond the ordinary cadence, even without lifecycle or replication.
Some(retry_interval.min(cycle_interval))
// A complete but non-authoritative walk must not postpone lifecycle or
// replication evaluation beyond its ordinary maintenance cadence.
if maintenance_features.needs_regular_cycle() {
retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1)));
}
Some(retry_interval)
}
/// Pacing for the cycle after a full-namespace scope fallback while dirty
+38 -73
View File
@@ -8949,7 +8949,7 @@ fn clean_idle_backoff_grows_to_cap() {
}
#[test]
fn retry_backoff_grows_caps_and_resets_after_convergence() {
fn superseded_retry_backoff_grows_caps_and_resets_after_convergence() {
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(backoff.retry_interval(Duration::from_secs(24 * 60 * 60)), None);
@@ -8973,7 +8973,7 @@ fn retry_backoff_grows_caps_and_resets_after_convergence() {
}
#[test]
fn retry_backoff_respects_a_faster_configured_cycle() {
fn superseded_retry_backoff_respects_a_faster_configured_cycle() {
let mut backoff = ScannerRetryBackoff::default();
backoff.record_retryable_cycle(true);
@@ -8985,7 +8985,7 @@ fn retry_backoff_respects_a_faster_configured_cycle() {
}
#[test]
fn retry_backoff_grows_from_the_default_cycle() {
fn superseded_retry_backoff_grows_from_the_default_cycle() {
let mut backoff = ScannerRetryBackoff::default();
// The first race after a write burst retries in seconds, not a whole
// cycle, while repeated supersedes still climb toward the cap.
@@ -9005,7 +9005,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &default_config),
scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()),
Some(SCANNER_RETRY_BASE_INTERVAL),
"the default adaptive cadence must retain fast convergence"
);
@@ -9015,24 +9015,24 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
ScannerRuntimeConfigSource::Config,
ScannerRuntimeConfigSource::ScannerCompatConfig,
] {
for cycle_secs in [1, 3, 600, 6 * 60 * 60] {
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
cycle_interval_source: source,
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(6 * 60 * 60),
cycle_interval_source: source,
..Default::default()
};
for features in [
ScannerMaintenanceFeatures::default(),
ScannerMaintenanceFeatures {
lifecycle: true,
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
for _ in 0..40 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must hold throughout convergence retries"
);
}
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
},
] {
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config, features),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must not be shortened by convergence retries"
);
}
}
}
@@ -9053,7 +9053,7 @@ fn superseded_retry_does_not_delay_required_maintenance() {
for consecutive_cycles in 1..=12 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &config),
scanner_superseded_retry_interval(backoff, &config, features),
Some(config.cycle_interval),
"superseded cycle {consecutive_cycles} must preserve regular maintenance"
);
@@ -9139,50 +9139,19 @@ fn scope_fallback_retry_preserves_explicit_and_fast_cycle_cadence() {
}
#[test]
fn superseded_retry_caps_at_the_default_cycle_and_resets_after_convergence() {
fn superseded_retry_only_caps_regular_maintenance() {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
for expected in [5, 10, 20, 40, 60, 60, 60, 60] {
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(Duration::from_secs(expected)));
}
backoff.consecutive_cycles = u32::MAX;
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
}
#[test]
fn superseded_retry_caps_fast_cycles_without_changing_the_generic_retry_cap() {
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..40 {
for _ in 0..6 {
backoff.record_retryable_cycle(true);
}
assert_eq!(
backoff.retry_interval(Duration::from_secs(60)),
Some(Duration::from_secs(30 * 60)),
"deferred and transient retries must retain the generic cap"
scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()),
Some(Duration::from_secs(160)),
);
for cycle_secs in [0, 1, 3, 60, 6 * 60 * 60] {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &config),
Some(Duration::from_secs(cycle_secs.max(1))),
);
}
}
#[test]
fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
for features in [
ScannerMaintenanceFeatures {
lifecycle: true,
@@ -9197,23 +9166,28 @@ fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
..Default::default()
},
] {
assert!(features.requires_full_scan(Some(7), 7, ScannerCycleWakeReason::Timer, true));
assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval));
assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None);
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle() {
async fn superseded_maintenance_wait_keeps_convergence_fencing() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let features = ScannerMaintenanceFeatures {
lifecycle: true,
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..12 {
backoff.record_retryable_cycle(true);
}
let interval = scanner_superseded_retry_interval(backoff, &config);
let interval = scanner_superseded_retry_interval(backoff, &config, features);
assert_eq!(interval, Some(config.cycle_interval));
let generations = ScannerCycleObservedGenerations::for_wait(
&config,
@@ -9228,28 +9202,19 @@ async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle()
let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]));
let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]);
let started = tokio::time::Instant::now();
let mut wait = Box::pin(wait_for_next_scanner_cycle_with_activity(
let reason = wait_for_next_scanner_cycle_with_activity(
&ctx,
interval.expect("superseded retry must retain its convergence interval"),
interval.expect("maintenance must retain its convergence interval"),
Some(Duration::from_secs(1)),
&mut seen,
generations,
|| false,
|| std::future::ready(Ok(changed.clone())),
));
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
for _ in 0..3 {
tokio::time::advance(Duration::from_secs(15)).await;
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
}
let reason = wait.await;
)
.await;
assert_eq!(reason, ScannerCycleWakeReason::Timer);
assert_eq!(started.elapsed(), config.cycle_interval);
assert_eq!(seen, Some(changed));
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[test]
+44 -105
View File
@@ -59,7 +59,6 @@ use rustfs_scanner_metrics::metrics::{
CloseDiskGuard, IlmAction, Metric, Metrics, ScannerReplicationRepairKind, ScannerSourceWorkUpdate, ScannerWorkSource,
UpdateCurrentPathFn, current_path_updater, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{ScannerCacheReuseDecision, ScannerCacheReuseGate, ScannerCacheReuseKind};
use rustfs_utils::path::{SLASH_SEPARATOR, path_join_buf};
use sha2::{Digest, Sha256};
use time::OffsetDateTime;
@@ -1061,51 +1060,38 @@ impl FolderScanner {
child_hash: &DataUsageHash,
abandoned_children: &DataUsageHashMap,
) -> bool {
let decision = if let Some(prefix_scan_scope) = &self.prefix_scan_scope {
if folder.parent.is_some() || folder.name != self.old_cache.info.name {
ScannerCacheReuseDecision::NotRoot
} else if into.compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !abandoned_children.contains(&child_hash.0) {
ScannerCacheReuseDecision::MissingCachedChild
} else if let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
{
if entry.is_empty() || entry.contains('/') {
ScannerCacheReuseDecision::InvalidPrefix
} else if prefix_scan_scope.contains(entry) {
ScannerCacheReuseDecision::DirtyPrefix
} else if self.is_erasure_mode {
// EC scans must still perform independent object-health work.
global_metrics().scan_observation().record_erasure_clean_prefix_walk();
ScannerCacheReuseDecision::ErasureHealthRequired
} else {
ScannerCacheReuseDecision::Eligible
}
} else {
ScannerCacheReuseDecision::InvalidPrefix
}
} else {
ScannerCacheReuseDecision::MissingScope
let Some(prefix_scan_scope) = &self.prefix_scan_scope else {
return false;
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, decision);
decision == ScannerCacheReuseDecision::Eligible
// Erasure-mode usage scans also perform probabilistic object-health
// work. Reusing a clean subtree here would silently suppress that
// independent maintenance path, so prefix reuse is limited to the
// non-erasure data-usage scanner.
if self.is_erasure_mode
|| folder.parent.is_some()
|| folder.name != self.old_cache.info.name
|| into.compacted
|| !abandoned_children.contains(&clone_cache_value(&child_hash.0))
{
return false;
}
let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
else {
return false;
};
!entry.is_empty() && !entry.contains('/') && !prefix_scan_scope.contains(entry)
}
fn reuse_clean_root_child(&mut self, child: &CachedFolder, child_hash: &DataUsageHash, into: &mut DataUsageEntry) {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
if let Some(updates) = &mut self.usage_updates {
updates.cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
}
into.add_child(child_hash);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, started.elapsed());
}
fn failed_retry_suppressed(&self, path: &str) -> bool {
@@ -1292,60 +1278,14 @@ impl FolderScanner {
}
fn can_reuse_covered_namespace(&self, name: &str, compacted: bool) -> bool {
let decision = if self.old_cache.info.scan_progress.is_none() {
ScannerCacheReuseDecision::NoResumeSweep
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
{
ScannerCacheReuseDecision::MissingCoverage
} else if self.old_cache.find(name).is_none() {
ScannerCacheReuseDecision::MissingCachedChild
} else if self.resume_subtree_has_failures(name) {
ScannerCacheReuseDecision::CachedFailures
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, decision);
decision == ScannerCacheReuseDecision::Eligible
}
// None preserves work that cannot use sampling; false requests this cycle's rescan.
fn sampled_namespace_reuse(&self, hash: &DataUsageHash, compacted: bool, forward_sweep: bool) -> Option<bool> {
let decision = if self.scan_mode == HealScanMode::Deep {
ScannerCacheReuseDecision::DeepScan
} else if forward_sweep {
ScannerCacheReuseDecision::ForwardSweep
} else if self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
{
ScannerCacheReuseDecision::ActiveReplication
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self.old_cache.is_compacted(hash) {
ScannerCacheReuseDecision::UncompactedChild
} else if hash.mod_(self.old_cache.info.next_cycle as u32, data_usage_update_dir_cycles()) {
ScannerCacheReuseDecision::SelectedForWalk
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::NamespaceSampling, decision);
match decision {
ScannerCacheReuseDecision::Eligible => Some(true),
ScannerCacheReuseDecision::SelectedForWalk => Some(false),
_ => None,
}
self.old_cache.info.scan_progress.is_some()
&& !compacted
&& self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
&& self.old_cache.find(name).is_some()
&& !self.resume_subtree_has_failures(name)
}
fn resume_subtree_has_failures(&self, folder: &str) -> bool {
@@ -2137,11 +2077,7 @@ impl FolderScanner {
// The caller stitches this cached child into its
// parent; do not clone a growing child set into the
// unused provisional accumulator as well.
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &this_hash, &folder.parent);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::ClassifiedNamespace, started.elapsed());
self.budget.record_entries_visited(pending_entry_progress);
self.finish_raw_enumeration_parent(&folder.name);
done_folder();
@@ -2728,12 +2664,8 @@ impl FolderScanner {
FolderScanMode::Namespace
};
if child_mode == FolderScanMode::Namespace && reuse_covered {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CoveredNamespace, started.elapsed());
continue;
}
@@ -2773,15 +2705,22 @@ impl FolderScanner {
}
FolderScanSource::Existing => {
// Usage sampling is not proof that a Deep check ran.
if let Some(reuse) = self.sampled_namespace_reuse(&h, into.compacted, forward_sweep) {
if reuse {
if self.scan_mode != HealScanMode::Deep
&& !forward_sweep
&& !self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
&& !into.compacted
&& self.old_cache.is_compacted(&h)
{
let next_cycle = self.old_cache.info.next_cycle as u32;
if !h.mod_(next_cycle, data_usage_update_dir_cycles()) {
if child_mode == FolderScanMode::Namespace {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::SampledNamespace, started.elapsed());
self.record_scan_resume_hint(&folder_item.name);
continue;
}
-191
View File
@@ -3103,112 +3103,6 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Directories));
}
#[tokio::test]
#[serial]
async fn cache_reuse_gates_report_only_the_first_production_condition() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
scanner.old_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
compacted: true,
objects: 2,
..Default::default()
},
);
let hash = hash_path("bucket/child");
let folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
let child = CachedFolder {
name: "bucket/child".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
};
let abandoned = HashSet::from([hash.0.clone()]);
let before = global_metrics().scan_observation().snapshot();
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let missing_scope = global_metrics().scan_observation().snapshot();
assert_eq!(
missing_scope.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"],
before.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"] + 1
);
scanner.prefix_scan_scope = ScannerBucketPrefixScanScope::from_dirty_top_level_entries(HashSet::from(["dirty".to_string()]));
scanner.is_erasure_mode = true;
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let erasure = global_metrics().scan_observation().snapshot();
assert_eq!(
erasure.reuse_decisions["clean_prefix"].blocked["erasure_health_required"],
before.reuse_decisions["clean_prefix"].blocked["erasure_health_required"] + 1
);
assert_eq!(erasure.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_frontier = Some("bucket/child".to_string());
scanner.resume_failed_paths = vec!["bucket/child/failed".to_string()];
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_failed_paths.clear();
assert!(scanner.can_reuse_covered_namespace("bucket/child", false));
let covered = global_metrics().scan_observation().snapshot();
let reasons = &covered.reuse_decisions["covered_namespace"];
let prior = &before.reuse_decisions["covered_namespace"];
assert_eq!(reasons.not_evaluated["no_resume_sweep"], prior.not_evaluated["no_resume_sweep"] + 1);
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["missing_coverage"], prior.blocked["missing_coverage"] + 1);
assert_eq!(reasons.blocked["cached_failures"], prior.blocked["cached_failures"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
with_var(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, Some("16"), || {
scanner.scan_mode = HealScanMode::Deep;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
scanner.scan_mode = HealScanMode::Normal;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
let replication: ReplicationConfig = serde_json::from_value(serde_json::json!({
"config": { "role": "", "rules": [{ "destination": { "bucket": "arn:test" }, "id": "status", "prefix": "", "priority": 1, "status": "Enabled" }] },
"remotes": crate::storage_api::EcstoreBucketTargets::default(),
})).expect("active replication fixture");
scanner.old_cache.info.replication = Some(Arc::new(replication));
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
scanner.old_cache.info.replication = None;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
let rescan_cycle = (0..16)
.find(|cycle| hash.mod_(*cycle, 16))
.expect("hash is visited in one sampling cycle");
scanner.old_cache.info.next_cycle = u64::from(rescan_cycle);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(false));
scanner.old_cache.info.next_cycle = u64::from((rescan_cycle + 1) % 16);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(true));
});
let after = global_metrics().scan_observation().snapshot();
let reasons = &after.reuse_decisions["namespace_sampling"];
let prior = &before.reuse_decisions["namespace_sampling"];
for label in ["deep_scan", "forward_sweep", "active_replication"] {
assert_eq!(
reasons.not_evaluated[label],
prior.not_evaluated[label] + 1,
"first sampling condition {label}"
);
}
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["selected_for_walk"], prior.blocked["selected_for_walk"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
for (kind, reuse) in &after.cache_reuse {
assert_eq!(reuse.count, before.cache_reuse[kind].count, "gate admission alone does not copy {kind}");
}
}
#[tokio::test]
#[serial]
async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entries() {
@@ -3250,21 +3144,12 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
.await
.expect("scoped root scan should finish successfully");
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count + 1);
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks);
assert_eq!(
observed.reuse_decisions["clean_prefix"].eligible,
before.reuse_decisions["clean_prefix"].eligible + 1
);
let clean = scanner
.new_cache
.size_recursive("bucket/clean")
@@ -3282,78 +3167,6 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
assert_eq!((bucket.size, bucket.objects), (17, 3));
}
#[tokio::test]
#[serial]
async fn scoped_cache_observations_distinguish_checkpoint_and_periodic_sampling() {
for covered in [true, false] {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
tokio::fs::create_dir_all(temp_dir.join("bucket/child"))
.await
.expect("create cached namespace");
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
size: 17,
objects: 5,
compacted: true,
..Default::default()
},
);
let kind = if covered {
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
scanner.resume_frontier = Some("bucket/child".to_string());
"covered_namespace"
} else {
let hash = hash_path("bucket/child");
let cycle = (0..data_usage_update_dir_cycles())
.find(|cycle| !hash.mod_(*cycle, data_usage_update_dir_cycles()))
.expect("a cycle samples other work");
scanner.old_cache.info.next_cycle = u64::from(cycle);
"sampled_namespace"
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.await
.expect("cached namespace reuses accounting");
let summary = scanner.new_cache.size_recursive("bucket").expect("complete bucket totals");
assert_eq!((summary.size, summary.objects), (17, 5));
let after = global_metrics().scan_observation().snapshot();
for label in [
"clean_prefix",
"covered_namespace",
"sampled_namespace",
"classified_namespace",
] {
assert_eq!(
after.cache_reuse[label].count,
before.cache_reuse[label].count + u64::from(label == kind),
"reuse reason {label}"
);
}
assert_eq!(after.metadata_reads.attempts, before.metadata_reads.attempts);
let gate = if covered { "covered_namespace" } else { "namespace_sampling" };
assert_eq!(after.reuse_decisions[gate].eligible, before.reuse_decisions[gate].eligible + 1);
}
}
async fn scan_hot_cold_segment_fixture(scoped: bool) -> (DataUsageEntry, Vec<String>) {
let (mut scanner, temp_dir) = build_test_scanner().await;
let visits = metadata_visits::observe(&mut scanner);
@@ -3465,7 +3278,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
@@ -3477,9 +3289,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
.size_recursive("bucket/clean")
.expect("erasure scan should visit the clean entry");
assert_eq!((clean.size, clean.objects), (0, 0));
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count);
}
#[tokio::test]
@@ -774,7 +774,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -791,15 +790,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
assert_eq!(budget.progress(), (0, 1), "only the parent consumes namespace quota");
assert_eq!(charged_ops(&root), 4, "list both directories once without re-reading covered metadata");
assert_eq!(scanner.new_cache.find(&folder.name).expect("reused candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
@@ -879,7 +869,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -900,15 +889,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
"sampling still requires full current classification, without metadata re-read"
);
assert_eq!(scanner.new_cache.find(&folder.name).expect("sampled candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
+46 -133
View File
@@ -40,7 +40,6 @@ use rustfs_lock::{LockError, NamespaceLockGuard};
use rustfs_scanner_metrics::metrics::{
Metric, Metrics, emit_scan_bucket_drive_complete, emit_scan_bucket_drive_partial, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{PREFIX_REUSE_BLOCKERS, ScannerBaselineRejection, ScannerPrefixReuseStage};
use rustfs_utils::path::path_join_buf;
use sha2::{Digest as _, Sha256};
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
@@ -113,7 +112,16 @@ pub(crate) const SCANNER_SEGMENT_ACTIVATION_PROOF_INPUTS: [&str; 9] = [
"durable_dirty_producer_journal",
"ack_generation_guard",
];
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = PREFIX_REUSE_BLOCKERS;
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
pub type DirtyUsageBuckets = HashMap<String, u64>;
@@ -189,7 +197,6 @@ struct VerifiedRemoteDirtyUsage {
}
struct ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Option<ScannerSegmentReuseActivationPreflight>,
scope: ScannerBucketScanScope,
remote_dirty_usage_acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
distributed_segment_invalidation_evidence: Option<DistributedSegmentInvalidationEvidence>,
@@ -229,13 +236,6 @@ pub(crate) struct ScannerSegmentReuseActivationPreflight {
}
impl ScannerSegmentReuseActivationPreflight {
fn record_observation(&self, stage: ScannerPrefixReuseStage) {
global_metrics().scan_observation().record_prefix_reuse(
stage,
self.scanner_segment_reuse_activated,
&self.fail_closed_blockers,
);
}
#[cfg(test)]
pub(crate) fn fail_closed_blockers(&self) -> impl Iterator<Item = &'static str> + '_ {
self.fail_closed_blockers.iter().filter_map(|blocker| *blocker)
@@ -319,7 +319,6 @@ fn resolve_remote_dirty_usage_scope(
baseline_proof: ScannerCacheBaselineProof<'_>,
) -> ScannerBucketScopeResolutionResult {
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -411,7 +410,6 @@ fn resolve_remote_dirty_usage_scope(
};
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
scope,
remote_dirty_usage_acknowledgements: scoped_acknowledgements,
distributed_segment_invalidation_evidence,
@@ -429,81 +427,43 @@ fn complete_scanner_cache_snapshot_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> Option<DataUsageScanPlanDigest> {
validate_scanner_cache_snapshot_plan_digest(snapshot, proof, expected_converged).ok()
}
fn validate_scanner_cache_snapshot_plan_digest(
snapshot: &DataUsageInfo,
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineRejection> {
// The observational mark disqualifies only the authoritative slot: an
// observed candidate carries it by construction.
let marks_accepted = match expected_converged {
Some(expected) => snapshot.usage_snapshot_converged == Some(expected),
None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational,
};
if !snapshot.is_complete_bucket_usage_snapshot() {
return Err(ScannerBaselineRejection::Incomplete);
}
if snapshot.usage_snapshot_partial {
return Err(ScannerBaselineRejection::Partial);
}
if !marks_accepted {
return Err(ScannerBaselineRejection::Marks);
}
if snapshot.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::Epoch);
}
if snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() {
return Err(ScannerBaselineRejection::SourceCount);
if !snapshot.is_complete_bucket_usage_snapshot()
|| snapshot.usage_snapshot_partial
|| !marks_accepted
|| snapshot.scanner_epoch != Some(proof.leader_epoch)
|| snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len()
{
return None;
}
// Completed maintenance also covers ordinary usage. Keep its exact stored
// proof for cache reuse, and reject mixtures of different set work proofs.
let baseline_plan_digest = DataUsageScanPlanDigest(
snapshot
.usage_snapshot_set_states
.first()
.and_then(|state| state.scan_plan_digest)
.ok_or(ScannerBaselineRejection::PlanMissing)?,
);
let baseline_plan_digest = DataUsageScanPlanDigest(snapshot.usage_snapshot_set_states.first()?.scan_plan_digest?);
if !scanner_plan_digest_is_work_variant(proof.scan_plan_digest, baseline_plan_digest) {
return Err(ScannerBaselineRejection::PlanMismatch);
return None;
}
let mut states = HashSet::with_capacity(snapshot.usage_snapshot_set_states.len());
for state in &snapshot.usage_snapshot_set_states {
let source = DataUsageCacheSource::new(
usize::try_from(state.pool_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
usize::try_from(state.set_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
);
if !proof.expected_sources.contains(&source) {
return Err(ScannerBaselineRejection::SourceMembership);
}
if !states.insert(source) {
return Err(ScannerBaselineRejection::SourceDuplicate);
}
if !state.complete {
return Err(ScannerBaselineRejection::SetIncomplete);
}
if state.tombstone {
return Err(ScannerBaselineRejection::SetTombstone);
}
if state.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::SetEpoch);
}
if state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle) {
return Err(ScannerBaselineRejection::SetCycle);
}
if state.scan_plan_digest != Some(baseline_plan_digest.0) {
return Err(ScannerBaselineRejection::SetPlan);
let source = DataUsageCacheSource::new(usize::try_from(state.pool_index).ok()?, usize::try_from(state.set_index).ok()?);
if !proof.expected_sources.contains(&source)
|| !states.insert(source)
|| !state.complete
|| state.tombstone
|| state.scanner_epoch != Some(proof.leader_epoch)
|| state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle)
|| state.scan_plan_digest != Some(baseline_plan_digest.0)
{
return None;
}
}
if states != *proof.expected_sources {
return Err(ScannerBaselineRejection::SourceCoverage);
}
Ok(baseline_plan_digest)
(states == *proof.expected_sources).then_some(baseline_plan_digest)
}
/// Whether `digest` is `structural` under any work flag: the same bucket
@@ -533,35 +493,10 @@ fn scanner_baseline_pending_buckets(proof: &ScannerCacheBaselineProof<'_>) -> Ve
}
fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option<DataUsageScanPlanDigest> {
validate_scanner_cache_baseline_plan_digest(proof).ok()
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection,
observed: Option<ScannerBaselineRejection>,
}
fn validate_scanner_cache_baseline_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineValidationFailure> {
let authoritative_error = |authoritative| ScannerBaselineValidationFailure {
authoritative,
observed: None,
};
let authoritative_data = proof
.authoritative_data
.ok_or_else(|| authoritative_error(ScannerBaselineRejection::Missing))?;
let authoritative = serde_json::from_slice::<DataUsageInfo>(authoritative_data)
.map_err(|_| authoritative_error(ScannerBaselineRejection::Decode))?;
let authoritative_rejection = match validate_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
Ok(validated_digest) => return Ok(validated_digest),
Err(reason) => reason,
};
let observed_error = |observed| ScannerBaselineValidationFailure {
authoritative: authoritative_rejection,
observed: Some(observed),
};
let authoritative = serde_json::from_slice::<DataUsageInfo>(proof.authoritative_data?).ok()?;
if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
return Some(validated_digest);
}
// A complete but superseded observation may reuse its per-set cache only
// when it was explicitly tied to the durable authoritative baseline. It
@@ -571,18 +506,14 @@ fn validate_scanner_cache_baseline_plan_digest(
&& !authoritative.usage_snapshot_observational)
|| crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative);
if !authoritative_has_identity {
return Err(observed_error(ScannerBaselineRejection::AuthoritativeIdentity));
return None;
}
let observed_data = proof
.observed_candidate_data
.ok_or_else(|| observed_error(ScannerBaselineRejection::Missing))?;
let observed =
serde_json::from_slice::<DataUsageInfo>(observed_data).map_err(|_| observed_error(ScannerBaselineRejection::Decode))?;
let observed = serde_json::from_slice::<DataUsageInfo>(proof.observed_candidate_data?).ok()?;
if !observed_data_usage_is_newer(&observed, &authoritative) {
return Err(observed_error(ScannerBaselineRejection::NotNewer));
return None;
}
validate_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)).map_err(observed_error)
complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false))
}
fn scanner_segment_reuse_baseline_producer_evidence(
@@ -727,12 +658,6 @@ impl ScannerScopeFallbackReason {
}
}
#[derive(Clone, Copy)]
struct ScannerScopeFallback {
reason: ScannerScopeFallbackReason,
baseline_rejection: Option<ScannerBaselineValidationFailure>,
}
/// Classify a resolved full-namespace scope against the preconditions
/// `resolve_scanner_bucket_scan_scope` checks, in the same order. Returns
/// `None` when the cycle was required to walk everything or was scoped.
@@ -743,24 +668,18 @@ fn scanner_scope_fallback_reason(
all_buckets: &[BucketInfo],
baseline_proof: ScannerCacheBaselineProof<'_>,
distributed: bool,
) -> Option<ScannerScopeFallback> {
let fallback = |reason| {
Some(ScannerScopeFallback {
reason,
baseline_rejection: None,
})
};
) -> Option<ScannerScopeFallbackReason> {
if !scope_eligible || !resolved_scope.is_default() {
return None;
}
if dirty_usage_snapshot.generation == u64::MAX {
return fallback(ScannerScopeFallbackReason::DirtyGenerationExhausted);
return Some(ScannerScopeFallbackReason::DirtyGenerationExhausted);
}
if !dirty_usage_snapshot.covers_listed_pending {
return fallback(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
return Some(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
}
if dirty_usage_snapshot.buckets.len() > crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES {
return fallback(ScannerScopeFallbackReason::DirtyBucketOverflow);
return Some(ScannerScopeFallbackReason::DirtyBucketOverflow);
}
// Peers may contribute dirty buckets the local snapshot does not hold.
if !distributed {
@@ -770,19 +689,13 @@ fn scanner_scope_fallback_reason(
.keys()
.any(|bucket| listed.contains(bucket.as_str()))
{
return fallback(ScannerScopeFallbackReason::NoListedDirtyBuckets);
return Some(ScannerScopeFallbackReason::NoListedDirtyBuckets);
}
}
if let Err(failure) = validate_scanner_cache_baseline_plan_digest(baseline_proof) {
return Some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::BaselineUnavailable,
baseline_rejection: Some(failure),
});
if complete_scanner_cache_baseline_plan_digest(baseline_proof).is_none() {
return Some(ScannerScopeFallbackReason::BaselineUnavailable);
}
distributed.then_some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::PeerScopeUnavailable,
baseline_rejection: None,
})
distributed.then_some(ScannerScopeFallbackReason::PeerScopeUnavailable)
}
fn scanner_segment_reuse_activation_preflight() -> ScannerSegmentReuseActivationPreflight {
+7 -11
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::*;
/// scanner cache locks and the cache snapshot persist/publish path.
use crate::data_usage_define::{DataUsageCacheBaselineRevisions, clone_cache_value};
use crate::data_usage_define::clone_cache_value;
use crate::{ScannerConfigObjectDelete, ScannerObjectIO};
use rustfs_scanner_metrics::usage_candidate::{
ScannerUsageCandidateKind as CandidateKind, ScannerUsageCandidateOutcome as CandidateOutcome,
@@ -1494,7 +1494,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
store: Arc<SetDisks>,
updates: &mpsc::Sender<DataUsageCache>,
mut cache_snapshot: DataUsageCache,
initial_revisions: Option<&DataUsageCacheBaselineRevisions>,
initial_revisions: Option<&DataUsageCacheRevisions>,
cache_cycle_floor: &AtomicU64,
expected_publication_epoch: u64,
) -> Option<SystemTime> {
@@ -1536,10 +1536,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
};
let mut persisted = DataUsageCache::default();
let current_baseline = match persisted
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
{
let revisions = match persisted.load_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME).await {
Ok(revisions) => revisions,
Err(err) => {
error!(
@@ -1561,7 +1558,6 @@ pub(super) async fn persist_and_publish_cache_snapshot(
return None;
}
};
let revisions = current_baseline.revisions();
let Some(scan_plan_digest) = cache_snapshot.info.scan_plan_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "plan_digest");
return None;
@@ -1625,9 +1621,9 @@ pub(super) async fn persist_and_publish_cache_snapshot(
{
cache_snapshot = persisted;
} else {
// Another execution must not advance the baseline. A backup catching
// up to the same decoded primary changes no baseline content.
if !initial_revisions.is_some_and(|initial| current_baseline.permits_unchanged_baseline_since(initial)) {
// A later execution may have completed while this scan was walking.
// Only replace the cache revision from which this scan started.
if initial_revisions != Some(&revisions) {
warn!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
@@ -1666,7 +1662,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
let done_save = Metrics::time(Metric::SaveUsage);
if let Err(e) = cache_snapshot
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, revisions, expected_publication_epoch)
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, &revisions, expected_publication_epoch)
.await
{
error!(
+3 -48
View File
@@ -14,7 +14,6 @@
use super::*;
/// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::{ScannerBucketReuseKind, ScannerScanObservation};
pub(super) async fn collect_completed_bucket_results(
ctx: CancellationToken,
@@ -143,32 +142,6 @@ pub(super) struct PreparedScopedSetScan {
pub(super) buckets: Vec<BucketInfo>,
pub(super) cache: DataUsageCache,
pub(super) cold_bucket_reuse_proof: Option<ScopedColdBucketReuseProof>,
cold_bucket_reuse_count: u64,
cold_bucket_reuse_elapsed: Duration,
}
impl PreparedScopedSetScan {
pub(super) fn record_cold_bucket_reuse(&self, observation: &ScannerScanObservation) {
observation.record_bucket_reuse(
ScannerBucketReuseKind::ScopedColdBucket,
self.cold_bucket_reuse_count,
self.cold_bucket_reuse_elapsed,
);
}
}
/// Counts only roots delivered to the collector; cancellation or channel failure is not reuse.
pub(super) async fn send_current_cache_root_entry(
bucket_result_tx: &mpsc::Sender<DataUsageEntryInfo>,
root: DataUsageEntryInfo,
cache: &DataUsageCache,
pending_maintenance_work: &AtomicBool,
observation: &ScannerScanObservation,
) -> std::result::Result<(), ScannerError> {
let started = Instant::now();
send_cache_root_entry(bucket_result_tx, root, cache, pending_maintenance_work).await?;
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, started.elapsed());
Ok(())
}
pub(super) struct ScopedColdBucketReuseProof {
@@ -252,8 +225,6 @@ pub(super) fn prepare_scoped_set_scan(
};
cache.replace(DATA_USAGE_ROOT, "", DataUsageEntry::default());
let mut current_bucket_names = HashSet::with_capacity(all_buckets.len());
let mut cold_bucket_reuse_count = 0u64;
let mut cold_bucket_reuse_elapsed = Duration::ZERO;
for bucket in all_buckets {
if !current_bucket_names.insert(bucket.name.as_str()) {
return None;
@@ -262,14 +233,11 @@ pub(super) fn prepare_scoped_set_scan(
cache.info.scan_bucket_value_origins.remove(&bucket.name);
cache.replace(&bucket.name, DATA_USAGE_ROOT, DataUsageEntry::default());
} else {
let started = Instant::now();
cache.copy_with_children(
old_cache,
&rustfs_data_usage::hash_path(&bucket.name),
&Some(rustfs_data_usage::hash_path(DATA_USAGE_ROOT)),
);
cold_bucket_reuse_count = cold_bucket_reuse_count.saturating_add(1);
cold_bucket_reuse_elapsed = cold_bucket_reuse_elapsed.saturating_add(started.elapsed());
}
}
@@ -281,8 +249,6 @@ pub(super) fn prepare_scoped_set_scan(
.collect(),
cache,
cold_bucket_reuse_proof,
cold_bucket_reuse_count,
cold_bucket_reuse_elapsed,
})
}
@@ -459,10 +425,7 @@ impl ScannerIOCache for SetDisks {
.ok_or_else(|| StorageError::other("scanner cache publication is blocked by data movement"))?,
};
let mut old_cache = DataUsageCache::default();
let initial_revisions = match old_cache
.load_with_baseline_revisions(self.clone(), DATA_USAGE_CACHE_NAME)
.await
{
let initial_revisions = match old_cache.load_with_revisions(self.clone(), DATA_USAGE_CACHE_NAME).await {
Ok(revisions) => Some(revisions),
Err(e) => {
warn!(
@@ -592,8 +555,6 @@ impl ScannerIOCache for SetDisks {
})
});
let mut scoped_cache = scoped_scan.map(|mut prepared| {
// The copies are completed work even if this turn later fails publication.
prepared.record_cold_bucket_reuse(global_metrics().scan_observation());
buckets = prepared.buckets;
prepared.cache.info.scan_coverage_digest = Some(bucket_coverage_digest);
prepared.cache
@@ -1597,14 +1558,8 @@ impl ScannerIOCache for SetDisks {
}
record_completed_usage_watch(&usage_watch_completions_clone, source, &bucket.name, bucket_usage_watch.as_ref()).await;
if let Err(e) =
send_current_cache_root_entry(
&bucket_result_tx_clone,
*root,
&cache,
&pending_maintenance_work_clone,
global_metrics().scan_observation(),
)
.await
send_cache_root_entry(&bucket_result_tx_clone, *root, &cache, &pending_maintenance_work_clone)
.await
{
record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await;
error!(
+3 -20
View File
@@ -357,7 +357,6 @@ where
S: ScannerStorage,
{
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -507,7 +506,7 @@ where
distributed,
resolution.baseline_proof,
);
let scope = scoped_scan_scope_from_dirty_buckets(
default_result(scoped_scan_scope_from_dirty_buckets(
resolution.requested_scope,
dirty_buckets,
(!distributed).then_some(resolution.dirty_usage_snapshot.scopes.as_ref()),
@@ -515,11 +514,7 @@ where
segment_reuse_activation_preflight.scanner_segment_reuse_activated,
resolution.all_buckets,
resolution.baseline_proof,
);
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
..default_result(scope)
}
))
}
/// Runs once between the bucket listing and the dirty-usage snapshot, the
@@ -764,9 +759,6 @@ where
},
)
.await;
if let Some(preflight) = scope_resolution.segment_reuse_activation_preflight {
preflight.record_observation(ScannerPrefixReuseStage::ScopeResolution);
}
let remote_dirty_usage_acknowledgements = scope_resolution.remote_dirty_usage_acknowledgements;
let distributed_segment_invalidation_evidence = scope_resolution.distributed_segment_invalidation_evidence;
let scan_scope = scope_resolution.scope;
@@ -778,14 +770,8 @@ where
baseline_proof,
distributed,
);
if let Some(fallback) = scope_fallback {
let reason = fallback.reason;
if let Some(reason) = scope_fallback {
global_metrics().record_scanner_scope_fallback(reason.as_str());
if let Some(failure) = fallback.baseline_rejection {
global_metrics()
.scan_observation()
.record_baseline_rejection(failure.authoritative, failure.observed);
}
tracing::info!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_SET_STATE,
@@ -799,7 +785,6 @@ where
"Scanner cycle walks every bucket because its dirty-bucket scope was not admitted"
);
}
let scope_fallback = scope_fallback.map(|fallback| fallback.reason);
let segment_invalidation_proof = dirty_usage_producer_evidence(&dirty_usage_snapshot).segment_invalidation_proof();
#[cfg(test)]
if let Some(observer) = resolved_scope_observer {
@@ -876,7 +861,6 @@ where
None,
false,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
return Ok(ScannerCycleResult::new(status, dirty_usage_clear)
.with_publication_epoch(publication_epoch)
.with_activity_digest(activity_digest)
@@ -1223,7 +1207,6 @@ where
distributed_segment_invalidation_evidence,
cold_zero_walk_oracle,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
let pending_maintenance_work = pending_maintenance_work_for_cycle(&pending_maintenance_work, &results);
let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire);
let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor);
+2 -15
View File
@@ -15,7 +15,6 @@ use super::*;
use crate::UNKNOWN_TIER;
/// ScannerIODisk implementation for Disk: get_size and the per-disk bucket scan.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::ScannerMetadataReadOutcome;
///
/// Seed [`SizeSummary::tier_stats`] from the cached tier-name list.
@@ -63,20 +62,8 @@ impl ScannerIODisk for Disk {
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let metadata_object_path = item.object_path();
let metadata_result = crate::workload_admission::with_scanner_metadata_io(async {
let read = global_metrics().start_scanner_metadata_read();
let result = self.read_metadata(&item.bucket, &metadata_object_path).await;
let (outcome, bytes) = match &result {
Ok(data) => (ScannerMetadataReadOutcome::Success, u64::try_from(data.len()).unwrap_or(u64::MAX)),
Err(e) if DiskError::is_err_object_not_found(e) || DiskError::is_err_version_not_found(e) => {
(ScannerMetadataReadOutcome::NotFound, 0)
}
Err(_) => (ScannerMetadataReadOutcome::Error, 0),
};
read.finish(outcome, bytes);
result
})
.await;
let metadata_result =
crate::workload_admission::with_scanner_metadata_io(self.read_metadata(&item.bucket, &metadata_object_path)).await;
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let data = match metadata_result {
Ok(data) => data,
+7 -558
View File
@@ -27,10 +27,10 @@ use crate::storage_api::owner::{
};
use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
use crate::{
BUCKET_META_PREFIX, DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints,
InstanceContext, PoolEndpoints, ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER,
init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx,
new_disk, path2_bucket_object_with_base_path, save_config,
DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
save_config,
};
use rustfs_concurrency::{
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
@@ -104,7 +104,6 @@ fn scanner_activity_preflight_defers_a_temporarily_offline_peer() {
#[test]
fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
let before = global_metrics().scan_observation().snapshot();
let preflight = scanner_segment_reuse_activation_preflight();
assert!(preflight.production_activation);
@@ -125,9 +124,6 @@ fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
"distributed_without_peer_invalidation",
]
);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated);
}
#[test]
@@ -2390,254 +2386,6 @@ async fn set_cache_send_failure_logs_receiver_and_snapshot_without_changing_retu
assert_set_cache_context(set_cache_event(&events, "publish_failed"), Some((0, 0)));
}
#[tokio::test]
#[serial]
async fn set_snapshot_accepts_backup_catchup_to_the_same_decoded_primary() {
use crate::data_usage_define::read_config_with_revision;
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read decoded primary bytes");
let primary_body = primary_body.expect("primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode older backup"))
.await
.expect("seed a lagging backup");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture primary with lagging backup");
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up without changing primary");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the reconciled backup");
assert_ne!(initial.revisions(), current.revisions());
assert!(current.permits_unchanged_baseline_since(&initial));
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
candidate.replace(
"photos",
DATA_USAGE_ROOT,
DataUsageEntry {
size: 20,
objects: 1,
..Default::default()
},
);
let cycle = candidate.info.next_cycle;
let cycle_floor = AtomicU64::new(cycle);
let (tx, mut rx) = mpsc::channel(1);
assert!(
persist_and_publish_cache_snapshot(Arc::clone(&set), &tx, candidate, Some(&initial), &cycle_floor, epoch)
.await
.is_some(),
"content-preserving backup catchup must not veto the new snapshot"
);
let published = rx.try_recv().expect("new snapshot reached the collector");
assert_eq!(published.info.next_cycle, cycle);
assert_eq!(published.find("photos").expect("new bucket entry").size, 20);
let (primary, primary_revision) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read new primary");
let (backup, backup_revision) = read_config_with_revision(set, &backup_path).await.expect("read new backup");
assert_eq!(primary, backup, "backup CAS must use the reconciled current revision");
assert_eq!(primary_revision, backup_revision);
let persisted = DataUsageCache::unmarshal(&primary.expect("new primary body")).expect("decode new primary");
assert_eq!(persisted.info.next_cycle, cycle);
assert_eq!(persisted.find("photos").expect("persisted new entry").size, 20);
}
#[tokio::test]
#[serial]
async fn set_snapshot_backup_catchup_keeps_revision_provenance_and_epoch_fences() {
use crate::data_usage_define::read_config_with_revision;
for case in ["different_backup", "different_primary", "corrupt_primary", "changed_epoch"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read original primary");
let primary_body = primary_body.expect("original primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode lagging backup"))
.await
.expect("seed lagging backup");
if case == "corrupt_primary" {
save_config(Arc::clone(&set), &primary_path, b"corrupt-primary".to_vec())
.await
.expect("inject primary corruption before the initial load");
}
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture original baseline source and revisions");
let mut replacement = cache.clone();
replacement.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
match case {
"different_backup" => {
save_config(Arc::clone(&set), &backup_path, replacement.marshal_msg().expect("encode changed backup"))
.await
.expect("backup changes to different contents");
}
"different_primary" => {
let body = replacement.marshal_msg().expect("encode changed primary");
save_config(Arc::clone(&set), &primary_path, body.clone())
.await
.expect("primary advances");
save_config(Arc::clone(&set), &backup_path, body)
.await
.expect("backup follows changed primary");
}
"corrupt_primary" => {
save_config(Arc::clone(&set), &backup_path, b"corrupt-primary".to_vec())
.await
.expect("backup matches the corrupt primary ETag without a decoded primary proof");
}
"changed_epoch" => {
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up before epoch refusal");
}
_ => unreachable!("fixed test cases"),
}
let before_primary = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("current primary");
let before_backup = read_config_with_revision(Arc::clone(&set), &backup_path)
.await
.expect("current backup");
let mut candidate = cache;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let cycle_floor = AtomicU64::new(candidate.info.next_cycle);
let requested_epoch = if case == "changed_epoch" {
epoch.checked_add(1).expect("fixture epoch can advance")
} else {
epoch
};
let (tx, mut rx) = mpsc::channel(1);
let (result, events) = capture_set_cache_logs(persist_and_publish_cache_snapshot(
Arc::clone(&set),
&tx,
candidate,
Some(&initial),
&cycle_floor,
requested_epoch,
))
.await;
assert!(result.is_none(), "{case} must still refuse publication");
assert!(
matches!(rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"{case} forwards no snapshot"
);
let expected_state = if case == "changed_epoch" {
"save_failed"
} else {
"scan_baseline_revision_changed"
};
assert_set_cache_context(set_cache_event(&events, expected_state), Some((0, 0)));
assert_eq!(
read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("protected primary"),
before_primary,
"{case} does not overwrite primary"
);
assert_eq!(
read_config_with_revision(set, &backup_path).await.expect("protected backup"),
before_backup,
"{case} does not overwrite backup"
);
}
}
#[tokio::test]
#[serial]
async fn set_cache_cas_preserves_updates_after_backup_catchup_validation() {
use crate::data_usage_define::read_config_with_revision;
for changed_copy in ["primary", "backup"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read initial primary");
let mut lagging = cache.clone();
lagging.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, lagging.marshal_msg().expect("lagging backup bytes"))
.await
.expect("seed backup lag");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read initial baseline");
save_config(Arc::clone(&set), &backup_path, primary_body.expect("initial primary exists"))
.await
.expect("finish backup catchup");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read reconciled baseline");
assert!(current.permits_unchanged_baseline_since(&initial));
let changed_path = if changed_copy == "primary" {
&primary_path
} else {
&backup_path
};
let mut concurrent = cache.clone();
concurrent.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
let concurrent_body = concurrent.marshal_msg().expect("encode concurrent update");
save_config(Arc::clone(&set), changed_path, concurrent_body.clone())
.await
.expect("copy changes after the baseline validation");
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let result = candidate
.save_with_revisions_for_epoch(Arc::clone(&set), DATA_USAGE_CACHE_NAME, current.revisions(), epoch)
.await;
if changed_copy == "primary" {
assert!(result.is_err(), "primary CAS refuses the intervening update");
} else {
result.expect("primary remains durable when the best-effort backup CAS refuses an update");
}
let (protected, _) = read_config_with_revision(Arc::clone(&set), changed_path)
.await
.expect("read protected concurrent copy");
assert_eq!(
protected,
Some(concurrent_body),
"{changed_copy} CAS must not overwrite the concurrent update"
);
let (primary, _) = read_config_with_revision(set, &primary_path)
.await
.expect("read resulting primary");
let primary = DataUsageCache::unmarshal(&primary.expect("primary exists")).expect("primary remains decodable");
assert_eq!(
primary.info.scan_execution_digest,
Some(DataUsageScanPlanDigest([if changed_copy == "primary" { 9 } else { 3 }; 32])),
);
}
}
#[tokio::test]
#[serial]
async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers() {
@@ -2652,7 +2400,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
.expect("seed legacy set cache");
let mut persisted = DataUsageCache::default();
let initial = persisted
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the shared starting revision");
let mut fresh = legacy.clone();
@@ -2678,7 +2426,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
assert_eq!(published.find("photos").expect("published bucket").size, 20);
assert_eq!(published.info.scan_execution_digest, fresh.info.scan_execution_digest);
let current = persisted
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the current revision for the unidentified execution");
@@ -2969,130 +2717,6 @@ fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provena
);
}
#[test]
fn scoped_scan_baseline_rejections_preserve_the_existing_first_failed_check() {
let source = DataUsageCacheSource::new(1, 2);
let expected_sources = HashSet::from([source]);
let digest = DataUsageScanPlanDigest([9; 32]);
let baseline = complete_usage_baseline(source, digest, 7, 11);
let snapshot: DataUsageInfo = serde_json::from_slice(&baseline).expect("decode complete baseline");
let proof = ScannerCacheBaselineProof {
authoritative_data: Some(&baseline),
observed_candidate_data: None,
expected_sources: &expected_sources,
leader_epoch: 11,
want_cycle: 8,
scan_plan_digest: digest,
};
type RejectionCase = (ScannerBaselineRejection, fn(&mut DataUsageInfo));
let cases: [RejectionCase; 13] = [
(ScannerBaselineRejection::Incomplete, |s| s.usage_snapshot_complete = false),
(ScannerBaselineRejection::Partial, |s| s.usage_snapshot_partial = true),
(ScannerBaselineRejection::Marks, |s| s.usage_snapshot_converged = None),
(ScannerBaselineRejection::Epoch, |s| s.scanner_epoch = Some(10)),
(ScannerBaselineRejection::SourceCount, |s| s.usage_snapshot_set_states.clear()),
(ScannerBaselineRejection::PlanMissing, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = None
}),
(ScannerBaselineRejection::PlanMismatch, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = Some([8; 32])
}),
(ScannerBaselineRejection::SourceMembership, |s| {
s.usage_snapshot_set_states[0].set_index = 3
}),
(ScannerBaselineRejection::SetIncomplete, |s| {
s.usage_snapshot_set_states[0].complete = false
}),
(ScannerBaselineRejection::SetTombstone, |s| {
s.usage_snapshot_set_states[0].tombstone = true
}),
(ScannerBaselineRejection::SetEpoch, |s| {
s.usage_snapshot_set_states[0].scanner_epoch = None
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = Some(9)
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = None
}),
];
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&snapshot, proof, None), Ok(digest));
for (expected, mutate) in cases {
let mut invalid = snapshot.clone();
mutate(&mut invalid);
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None), Err(expected));
assert_eq!(complete_scanner_cache_snapshot_plan_digest(&invalid, proof, None), None);
}
let mut invalid = snapshot.clone();
invalid.usage_snapshot_partial = true;
invalid.scanner_epoch = Some(10);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None),
Err(ScannerBaselineRejection::Partial)
);
let two_sources = HashSet::from([source, DataUsageCacheSource::new(1, 3)]);
let mut two_sets = snapshot.clone();
two_sets
.usage_snapshot_set_states
.push(two_sets.usage_snapshot_set_states[0].clone());
let two_set_proof = ScannerCacheBaselineProof {
expected_sources: &two_sources,
..proof
};
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SourceDuplicate)
);
two_sets.usage_snapshot_set_states[1].set_index = 3;
two_sets.usage_snapshot_set_states[1].scan_plan_digest = Some([8; 32]);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SetPlan)
);
let undecodable = Bytes::from_static(b"not-json");
for (data, expected) in [
(None, ScannerBaselineRejection::Missing),
(Some(&undecodable), ScannerBaselineRejection::Decode),
] {
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
authoritative_data: data,
..proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: expected,
observed: None
})
);
}
let mut wrong_epoch = snapshot;
wrong_epoch.scanner_epoch = Some(10);
let wrong_epoch = Bytes::from(serde_json::to_vec(&wrong_epoch).expect("encode stale epoch baseline"));
let stale_proof = ScannerCacheBaselineProof {
authoritative_data: Some(&wrong_epoch),
..proof
};
assert_eq!(
validate_scanner_cache_baseline_plan_digest(stale_proof),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Missing),
})
);
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
observed_candidate_data: Some(&undecodable),
..stale_proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Decode)
})
);
}
#[test]
fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_baseline() {
let source = DataUsageCacheSource::new(1, 2);
@@ -3526,12 +3150,6 @@ fn distributed_segment_reuse_activation_keeps_remote_dirty_buckets_at_bucket_sco
result.scope.prefix_scope_for("photos").is_some(),
"durable local producer proof may activate local segment reuse after distributed ACK capability evidence"
);
assert!(
result
.segment_reuse_activation_preflight
.expect("actual scope proof is retained for observation")
.scanner_segment_reuse_activated
);
assert!(
result.scope.prefix_scope_for("archive").is_none(),
"remote dirty usage has only bucket-granularity evidence and must not be narrowed to a local prefix"
@@ -3950,10 +3568,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
.checked_flatten("stable")
.expect("selected bucket placeholder should exist");
assert_eq!((stable.size, stable.objects), (0, 0));
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
assert!(prepared.cache.find("stable/prefix").is_none());
assert_eq!(prepared.cache.find("dirty").map(|entry| (entry.size, entry.objects)), Some((0, 0)));
assert!(prepared.cache.find("deleted").is_none());
@@ -3965,124 +3579,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
assert_eq!(prepared.cache.info.lkg_scan_plan_digest, Some(baseline_digest));
}
#[test]
fn cold_bucket_reuse_observation_rejects_unbound_or_failed_preparation() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
let mut old_cache = complete_set_usage_cache(&[("stable", 10), ("dirty", 20)], baseline_digest);
old_cache.info.scan_bucket_incarnations = test_bucket_incarnations(&["stable", "dirty"]);
let incarnations = old_cache.info.scan_bucket_incarnations.clone();
let buckets = vec![
bucket_info_with_created_time("stable"),
bucket_info_with_created_time("dirty"),
];
let scope = ScannerBucketScanScope {
selected_buckets: Some(Arc::new(HashSet::from(["dirty".to_string()]))),
selected_bucket_prefixes: None,
baseline_scan_plan_digest: Some(baseline_digest),
};
let generation = ScannerSetCacheGeneration {
want_cycle: 8,
leader_epoch: 11,
tier_registry_generation: 13,
source: DataUsageCacheSource::new(1, 2),
scan_plan_digest: DataUsageScanPlanDigest([2; 32]),
};
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
let prepared = prepare_scoped_set_scan(&old_cache, &buckets, &buckets, &scope, generation, Some(&incarnations))
.expect("bound complete cache permits a cold bucket copy");
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 1);
for case in ["incomplete", "duplicate", "nil_incarnation"] {
let mut invalid_cache = old_cache.clone();
let mut invalid_buckets = buckets.clone();
let mut invalid_incarnations = incarnations.clone();
match case {
"incomplete" => invalid_cache.info.snapshot_complete = false,
"duplicate" => invalid_buckets.push(buckets[0].clone()),
"nil_incarnation" => {
invalid_incarnations.insert("stable".to_string(), Uuid::nil());
}
_ => unreachable!("fixed test cases"),
}
let prepared = prepare_scoped_set_scan(
&invalid_cache,
&invalid_buckets,
&invalid_buckets,
&scope,
generation,
Some(&invalid_incarnations),
);
assert!(prepared.is_none(), "{case} must not return a reusable prepared cache");
if let Some(prepared) = prepared {
prepared.record_cold_bucket_reuse(&observation);
}
assert_eq!(
observation.snapshot().bucket_reuse["scoped_cold_bucket"].count,
1,
"{case} does not count completed reuse"
);
}
}
#[tokio::test]
async fn current_root_reuse_counts_only_successful_collector_delivery() {
use rustfs_scanner_metrics::scan_observation::ScannerScanObservation;
let observation = ScannerScanObservation::default();
let cache = DataUsageCache::default();
let pending = AtomicBool::new(false);
let root = || DataUsageEntryInfo {
name: "bucket".to_string(),
parent: DATA_USAGE_ROOT.to_string(),
entry: DataUsageEntry {
size: 10,
objects: 2,
..Default::default()
},
bucket_incarnation: Some(Uuid::from_u128(7)),
tier_registry_generation: None,
value_origin: None,
};
let (sender, mut receiver) = mpsc::channel(1);
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.expect("current root is delivered to the collector");
let delivered = receiver.recv().await.expect("collector received the root");
assert_eq!((delivered.entry.size, delivered.entry.objects), (10, 2));
assert_eq!(observation.snapshot().bucket_reuse["current_root"].count, 1);
drop(receiver);
assert!(
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.is_err()
);
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"closed collector is not completed reuse"
);
let (sender, _receiver) = mpsc::channel(1);
sender.send(root()).await.expect("fill bounded collector channel");
{
let mut send = Box::pin(super::io_cache::send_current_cache_root_entry(
&sender,
root(),
&cache,
&pending,
&observation,
));
assert!(futures::poll!(send.as_mut()).is_pending(), "full collector keeps reuse unfinished");
}
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"dropping unfinished delivery is not reuse"
);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
}
#[test]
fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
@@ -4124,12 +3620,6 @@ fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
)
.expect("matching bucket incarnations should authorize cold bucket reuse");
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
let observed = observation.snapshot();
assert_eq!(observed.bucket_reuse["scoped_cold_bucket"].count, 1);
assert_eq!(observed.bucket_reuse["current_root"].count, 0);
assert!(observed.cache_reuse.values().all(|reuse| reuse.count == 0));
assert_eq!(prepared.buckets.iter().map(|bucket| bucket.name.as_str()).collect::<Vec<_>>(), ["dirty"]);
let stable = prepared
.cache
@@ -5135,36 +4625,12 @@ async fn get_size_treats_missing_metadata_as_skip_file() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let err = disk
.get_size(item.clone())
.get_size(item)
.await
.expect_err("missing metadata should be skipped instead of reported as a scanner failure");
assert!(matches!(err, StorageError::Io(ref io) if io.to_string() == SCANNER_SKIP_FILE_ERROR));
let missing = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(missing.attempts, before.attempts + 1);
assert_eq!(missing.not_found, before.not_found + 1);
assert_eq!(missing.bytes, before.bytes);
#[cfg(unix)]
{
tokio::fs::create_dir_all(&object_dir)
.await
.expect("recreate object directory");
std::os::unix::fs::symlink(STORAGE_FORMAT_FILE, &metadata_path).expect("inject a metadata symlink loop");
let err = disk
.get_size(item)
.await
.expect_err("symlink loop must remain a read failure");
assert!(is_scanner_metadata_transient_error(&err), "read error: {err:?}");
let failed = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(failed.attempts, missing.attempts + 1);
assert_eq!(failed.errors, missing.errors + 1);
assert_eq!(failed.succeeded, before.succeeded);
assert_eq!(failed.bytes, before.bytes);
}
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -5215,7 +4681,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
for contents in [b"".as_slice(), b"not-valid-filemeta".as_slice()] {
tokio::fs::write(&metadata_path, contents)
.await
@@ -5227,16 +4692,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
assert!(is_scanner_metadata_corrupt_error(&err));
}
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 2);
assert_eq!(
reads.succeeded,
before.succeeded + 2,
"storage reads succeed before decoding rejects corruption"
);
assert_eq!(reads.errors, before.errors);
assert_eq!(reads.bytes, before.bytes + 18);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -5309,17 +4764,11 @@ async fn get_size_counts_delete_markers_separately_from_versions() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let expected_bytes = tokio::fs::metadata(&metadata_path).await.expect("stat stored metadata").len();
let summary = disk.get_size(item).await.expect("scanner should read versioned metadata");
assert_eq!(summary.versions, 2);
assert_eq!(summary.delete_markers, 1);
assert_eq!(summary.total_size, 30);
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 1);
assert_eq!(reads.succeeded, before.succeeded + 1);
assert_eq!(reads.bytes, before.bytes + expected_bytes);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -183,11 +183,9 @@ async fn dirty_entry_removed_during_listing_still_forces_a_full_scan() {
);
assert_eq!(second.buckets_usage["hot-bucket"].objects_count, 2);
assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2);
let report = global_metrics().report().await;
let freshness = report.usage_freshness;
let freshness = global_metrics().report().await.usage_freshness;
assert_eq!(freshness.scope_fallback_total, fallbacks_before + 1);
assert_eq!(freshness.last_scope_fallback_reason, "dirty_snapshot_incomplete");
assert_eq!(report.scan_observation.scope_fallbacks["dirty_snapshot_incomplete"], 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -201,53 +199,11 @@ async fn required_full_scan_is_not_reported_as_a_scope_fallback() {
wait_for_namespace_commit_tails(store.as_ref()).await;
super::io_cycle::set_bucket_listing_window_hook_for_tests(|| record_dirty_usage_bucket("created-after-listing"));
let before = global_metrics().scan_observation().snapshot();
let (result, _, scope) = run_cycle(&store, 2, Some(baseline), true).await;
assert!(scope.is_default());
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert_eq!(result.scope_fallback(), None, "scheduled maintenance walks are not degradations");
assert_eq!(result.completed_scope_fallback(), None);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.scope_fallbacks, before.scope_fallbacks);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn baseline_unavailable_cycle_records_real_scope_and_baseline_rejections_once() {
let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
let _baseline = converged_baseline(&store).await;
put_fixture_object(&store, "hot-bucket", "second").await;
record_dirty_usage_bucket("hot-bucket");
wait_for_namespace_commit_tails(store.as_ref()).await;
let before = global_metrics().scan_observation().snapshot();
let (result, snapshot, scope) = run_cycle(&store, 2, Some(Bytes::from_static(b"not-json")), false).await;
assert!(scope.is_default());
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(result.scope_fallback(), Some(ScannerScopeFallbackReason::BaselineUnavailable));
assert_eq!(snapshot.buckets_usage["hot-bucket"].objects_count, 2);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(
after.scope_fallbacks["baseline_unavailable"],
before.scope_fallbacks["baseline_unavailable"] + 1
);
assert_eq!(
after.authoritative_baseline_rejections["decode"],
before.authoritative_baseline_rejections["decode"] + 1
);
assert_eq!(
after.observed_baseline_rejections, before.observed_baseline_rejections,
"an undecodable authority never admits an observation"
);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated + 1);
assert_eq!(after.prefix_scope_resolution.activated, before.prefix_scope_resolution.activated);
assert_eq!(
after.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"],
before.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"] + 1
);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -1,12 +1,12 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! A bucket whose walk stops early can publish its last certified value as
//! pending. This fixture tests usage-scan status, not the outer scanner cycle
//! outcome: budget exhaustion may leave complete validated usage while the
//! coordinator records `Partial`. A continuation under a newer plan exercises
//! `mixed_coverage` (rustfs/rustfs#8494). A bucket that never certified keeps
//! the usage scan incomplete.
//! A bucket whose walk stops early without exhausting the cycle budget (here
//! a sweep resumed under a newer plan, the `mixed_coverage` case of
//! rustfs/rustfs#8494) no longer makes the whole cycle `Incomplete`. The set
//! publishes the bucket's last certified value, the cycle completes with the
//! bucket pending, and the next cycle walks it again. A bucket that never
//! certified keeps the cycle incomplete.
use super::*;
@@ -38,7 +38,8 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
..Default::default()
},
);
// Real checkpoint commits need the runtime drain grace after the count budget stops the walk.
// These cycles persist real bucket checkpoints and source caches. Keep
// CI storage latency inside the production drain window after a count stop.
budget.use_production_drain_grace_for_tests();
let (updates, mut receiver) = mpsc::channel(1);
let result = tokio::time::timeout(
@@ -48,7 +49,7 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
ScannerCycleRequest {
owner_ctx: ctx.clone(),
ctx: budget.token(),
budget: Arc::clone(&budget),
budget,
updates,
want_cycle: cycle,
leader_epoch: LEADER_EPOCH,
@@ -68,13 +69,6 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
.await
.expect("cycle should finish within the test deadline")
.expect("cycle should succeed");
if options.max_objects.is_some() {
assert_eq!(
budget.reason(),
Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects),
"a count-limited fixture must stop for its object budget"
);
}
let snapshot = receiver.recv().await;
assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot");
(result, snapshot)
@@ -131,8 +125,9 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
assert!(result.clear_verified_usage().is_empty());
assert!(dirty_usage_buckets_for_tests().is_empty());
// Cycle 3: the budget stops the walk and saves a checkpoint. Usage can
// carry the certified value as pending; the outer coordinator records Partial.
// Cycle 3: the object budget stops the hot bucket's walk and leaves a
// checkpoint under this cycle's plan. Its certified value is carried, so
// budget exhaustion leaves the bucket pending without blocking the cycle.
for object in ["p4/third", "p5/third"] {
put_fixture_object(&store, "hot-bucket", object).await;
}
@@ -149,38 +144,12 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
},
)
.await;
assert_eq!(
result.status,
ScannerCycleStatus::Complete,
"certified carried usage remains complete after a budget stop"
);
let third = third.expect("validated carried usage publishes after the budget stop");
let third = third.expect("a carried partial cycle publishes");
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert!(!third.usage_snapshot_observational);
assert_eq!(third.usage_snapshot_converged, Some(false));
assert_eq!(third.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]);
assert_eq!(third.usage_snapshot_pending_bucket_count, 1);
assert!(third.is_complete_bucket_usage_snapshot());
let carried_hot = &third.buckets_usage["hot-bucket"];
let certified_hot = &second.buckets_usage["hot-bucket"];
assert_eq!(carried_hot.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Carried));
assert!(certified_hot.usage_certification.is_some(), "cycle 2 certified the prior hot value");
assert_eq!(
carried_hot.usage_certification, certified_hot.usage_certification,
"carrying must not mint a new certificate"
);
assert_eq!(carried_hot.objects_count, 4);
assert_eq!(carried_hot.size, certified_hot.size);
assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 1);
assert_eq!(third.buckets_usage["idle-bucket"].objects_count, 1);
assert_eq!(third.objects_total_count, 6);
// Only hot was dirty in this scoped scan; it received no fresh certification or ACK.
assert!(result.clear_verified_usage().is_empty());
assert_eq!(
dirty_usage_buckets_for_tests().keys().cloned().collect::<Vec<_>>(),
vec!["hot-bucket".to_string()],
"the budget-stopped hot bucket retains its dirty responsibility"
);
assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 4);
let set = Arc::clone(&store.pools[0].disk_set[0]);
let mut interrupted = DataUsageCache::default();
interrupted
+1 -1
View File
@@ -109,7 +109,7 @@ pub use bucket::{BucketInfo, BucketOperations, BucketOptions, DeleteBucketOption
pub use capability::{CapabilitySnapshotError, CapabilityState, CapabilityStatus};
pub use error::{StorageErrorCode, StorageResult};
pub use multipart::{CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartUploadResult, PartInfo};
pub use multipart::{MultipartCompletionAuthorization, MultipartUploadInspection};
pub use multipart::{MultipartAbortAuthorization, MultipartCompletionAuthorization, MultipartOwner, MultipartUploadInspection};
pub use object::AuthoritativeListOperations;
pub use object::DeleteAccounting;
pub use object::ObjectLockDeleteOptions;
+28
View File
@@ -15,6 +15,24 @@
use std::collections::HashMap;
use time::OffsetDateTime;
/// Ownership observed in the selected upload metadata.
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct MultipartOwner {
/// The bucket recorded by the upload.
pub bucket: String,
/// The recorded object, decoded after checking the encoded storage namespace.
pub object: String,
}
/// Authorizes Abort against the upload selected under its existing write lock.
/// An implementation must reject missing ownership and must not acquire storage locks or mutate storage.
pub trait MultipartAbortAuthorization: Send + Sync {
type Error: std::error::Error + Send + Sync + 'static;
/// Returns a refusal before storage prepares or deletes any staged data.
fn authorize(&self, owner: Option<&MultipartOwner>) -> Result<(), Self::Error>;
}
/// Request authorization against the source selected under completion locks.
/// Implementations must not mutate storage or acquire object/upload locks again.
/// Called before comparing client checksums/parts or acknowledging a replay.
@@ -73,8 +91,18 @@ pub struct ListMultipartsInfo {
pub common_prefixes: Vec<String>,
}
/// The owner found in the upload metadata used to read a ListParts page.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ListPartsOwner {
pub bucket: String,
/// The recorded object after decoding the storage namespace spelling.
pub object: String,
}
#[derive(Debug, Clone, Default)]
pub struct ListPartsInfo {
/// Present when the recorded bucket and encoded object agree with the storage namespace.
pub recorded_owner: Option<ListPartsOwner>,
pub bucket: String,
pub object: String,
pub upload_id: String,
@@ -60,6 +60,7 @@ Run **before** at the baseline column and **after** at the enabled column, every
| P1 isolation | `RUSTFS_INTERNODE_CHANNEL_ISOLATION` | `false` (default) | `true` |
| P1 bulk pool | `RUSTFS_INTERNODE_BULK_CHANNELS` | unset | `2` |
| P2 msgpack-only | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` + `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` | both `false` (default) | per the convergence runbook |
| P2 per-peer msgpack | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` + `RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY` | both `false` (default) | both `true`, fleet confirmation `false`, stable direct/affine peer connections only |
| P3 prewarm | `RUSTFS_INTERNODE_PREWARM` | `false` (default) | `true` |
| P3 offline bypass | `RUSTFS_INTERNODE_OFFLINE_BYPASS` | `false` (default) | `true` |
| P3 reprobe / threshold | `RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS` / `RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD` | defaults | defaults (`DEFAULT_INTERNODE_OFFLINE_REPROBE_SECS`, `DEFAULT_INTERNODE_OFFLINE_FAILURE_THRESHOLD`) |
@@ -1,7 +1,7 @@
# Internode msgpack/JSON convergence runbook
**Use this when:** operating or changing the staged retirement of the JSON compatibility fields on internode gRPC metadata RPCs, deciding whether a fleet may enable `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY`, or adding a new `*_bin` proto field.
**Source of truth:** `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_RPC_MSGPACK_ONLY`, `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` and their compile-time default-off asserts), `crates/protos/src/node.proto` (`*_bin` fields), `decode_msgpack_or_json` in `rustfs/src/storage/rpc/node_service/disk.rs` (server side) and `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client side), `crates/io-metrics/src/internode_metrics.rs` (counters).
**Source of truth:** `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_RPC_MSGPACK_ONLY`, `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED`, `ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY` and their compile-time default-off asserts), `crates/protos/src/node.proto` (`*_bin` fields), `decode_msgpack_or_json` in `rustfs/src/storage/rpc/node_service/disk.rs` (server side) and `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client side), `crates/io-metrics/src/internode_metrics.rs` (counters).
This is a cross-version change and proceeds by observation-gated stages, never in one step.
@@ -16,6 +16,7 @@ Internode RPCs dual-encode each metadata value as a msgpack binary field (`*_bin
| `rustfs_system_network_internode_msgpack_json_decode_total` | `direction`, `message`, `codec` | a msgpack or JSON decode succeeded |
| `rustfs_system_network_internode_msgpack_json_fallback_total` | `direction`, `message` | a decode fell back to JSON because `_bin` was empty |
| `rustfs_system_network_internode_msgpack_json_decode_error_total` | `direction`, `message`, `codec` | either codec failed to decode (`codec` = the failed codec) |
| `rustfs_system_network_internode_msgpack_send_decisions_total` | `decision`, `reason` | an eligible RPC selected `msgpack_only` or `dual_write`; reasons are fixed to `manual_fleet`, `peer_capable`, `peer_unknown`, `peer_revoked`, or `not_requested` |
`direction="request"` is a server decoding a peer's request (`node_service/disk.rs`); `direction="response"` is a client decoding a peer's response (`cluster/rpc/remote_disk.rs`), including the list-level `ReadMultiple` / `BatchReadVersion` fallbacks. `message` is the value name (`FileInfo`, `RawFileInfo`, `ReadMultipleResp`, ...).
@@ -92,15 +93,55 @@ Run the fleet with both flags at their default (`false`) for at least one full o
## Stage 1 — Stop writing JSON (env-gated)
Flag semantics are defined on `ENV_INTERNODE_RPC_MSGPACK_ONLY` and `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` in `crates/config/src/constants/internode.rs`; both default off and only their conjunction stops the JSON write for convergence-ready fields. Decoders keep the JSON fallback in every state.
Flag semantics are defined in `crates/config/src/constants/internode.rs`; all three flags default off. `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` requests msgpack-only, and JSON stops only when either fleet confirmation is true or the peer-capability gate is true and the exact current peer boot epoch plus physical connection generation has an authenticated capability proof. Unknown, unsupported, revoked, stale, or reconnecting peers continue dual-write. Decoders keep the JSON fallback in every state.
1. Rehearse `request-only`: `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true`, `..._FLEET_CONFIRMED=false`. Behaviour is unchanged (still dual-write) and safe with old peers; any fallback or decode-error increment still blocks.
2. Canary: set both flags `true` on one node, restart that node only, and watch the fallback and decode-error counters for a soak period with real internode traffic.
3. Fleet: if the counters stay zero, enable both flags fleet-wide with a rolling restart.
4. Rollback: set either flag to `false` (or unset it) and restart; no wire format changed, so rollback is immediate.
An alternative per-peer canary keeps `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false`, sets `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY=true`, and confirms `reason="peer_capable"` only for proved peers while every `peer_unknown` or `peer_revoked` decision remains `dual_write`. Enable this mode only when one client HTTP/2 connection has stable affinity to one RustFS peer process; an L7 proxy that routes streams from one connection to different backend versions invalidates the connection-generation proof. A connection-generation change may rebuild one request as dual-write only when the retained local guard proves rejection before the first payload frame; remote status text, a dispatched request, or an ordinary transport error never authorizes that compatibility retry. Disable either the base request flag or peer-capability flag to roll back immediately.
The benchmark driver pins these operator states as dry-run phases (`before`, `request-only`, `canary`, `after`, `rollback`); see [internode-grpc-benchmark-runbook.md](internode-grpc-benchmark-runbook.md).
### Per-peer mixed-version evidence gate
The per-peer gate is not qualified by a successful command transcript alone. Record three ordered four-node measurement phases: `mixed-old-peer` (three candidate nodes and one previous-version node), `all-new` (four candidate nodes), and `rollback-old-peer` (replace one candidate with the previous-version image and verify readback). Every candidate node uses `MSGPACK_ONLY=true`, `MSGPACK_PEER_CAPABILITY=true`, and `MSGPACK_ONLY_FLEET_CONFIRMED=false`; previous-version nodes use all three false. Each phase must run real PUT/GET/metadata traffic and retain the exact candidate/baseline revisions, topology, timestamps, workload result, environment, and raw Prometheus HTTP API JSON.
Assign one unique `msgpack-<32-hex>` cluster ID for this run, set `OTEL_RESOURCE_ATTRIBUTES=rustfs.cluster.id=<id>` on every node, and retain the collector’s resource-to-telemetry conversion so Prometheus exposes `rustfs_cluster_id`. Capture these PromQL vectors over each non-overlapping phase window and store the complete HTTP API response, not extracted numbers:
```promql
sum by (decision, reason) (increase(rustfs_system_network_internode_msgpack_send_decisions_total{rustfs_cluster_id="$CLUSTER_ID"}[$WINDOW]))
sum(increase(rustfs_system_network_internode_msgpack_json_fallback_total{rustfs_cluster_id="$CLUSTER_ID"}[$WINDOW])) or vector(0)
sum(increase(rustfs_system_network_internode_msgpack_json_decode_error_total{rustfs_cluster_id="$CLUSTER_ID"}[$WINDOW])) or vector(0)
sum(increase(rustfs_system_network_internode_operation_errors_total{rustfs_cluster_id="$CLUSTER_ID"}[$WINDOW])) or vector(0)
```
Build a schema-1 manifest with `scope="internode-msgpack-peer-canary"`, `source_revision`, `baseline_revision`, `stable_connection_affinity=true`, `max_metric_lag_secs`, `cluster_id`, an `images` path to sanitized image inspections, and the three phase records. Each phase records integer `start_unix`/`end_unix`, exact topology counts plus a `node1..node4` revision map, the new/old node environment maps, the exact four PromQL strings with a window equal to the phase duration, an `images` path to the verified per-node image ID/revision map, a relative workload JSON path, and relative paths for `decisions`, `fallback`, `decode_errors`, and `rpc_errors`. The workload artifact records `status`, a non-zero integer `requests`, and `rollback_readback` (true only for `rollback-old-peer`). Validate without rewriting raw evidence:
```bash
python3 scripts/check_internode_msgpack_peer_canary.py \
--manifest /path/to/evidence/manifest.json \
--out /path/to/evidence/verification.json
```
The verifier rejects reused/escaping/overwritten artifacts, duplicate or non-exact decision labels, labeled scalar queries, mismatched PromQL windows, wrong per-node revisions, missing peer-capable traffic, missing mixed/rollback dual-write traffic, manual-fleet or not-requested leakage, failed workload/readback, non-zero fallback/decode/RPC errors, stale timestamps, invalid topology, and candidate/baseline identity reuse. The workload artifact binds its raw benchmark `summary.csv` by relative path and SHA-256; rollback also records equal source/download content hashes. Verifier output includes SHA-256 receipts for every raw query, workload, and summary artifact. Passing this gate is compatibility evidence only; production soak and ABBA performance drift gates remain separate.
Use the orchestration runner to create the isolated Compose project, execute the three phases, preserve a known object across node2 rollback, capture the four raw Prometheus responses, build the manifest, and invoke the verifier. Without `--execute` it prints a redacted plan and makes no external changes. The runner verifies distinct immutable image IDs and full `vcs-ref` or OCI revision labels against the requested revisions, pins Compose to those IDs, and checks each running container’s image and telemetry identity in every phase. Images without matching full revision labels are rejected. It retains sanitized identity receipts without container credentials. Execution refuses an existing Compose project/volume set and removes its isolated containers and volumes on exit unless `--keep-up` is explicit.
```bash
scripts/run_internode_msgpack_peer_canary.sh \
--candidate-image rustfs/rustfs:candidate \
--baseline-image rustfs/rustfs:previous \
--candidate-revision <40-hex-candidate-commit> \
--baseline-revision <40-hex-previous-commit> \
--prometheus-query-url http://127.0.0.1:9090/api/v1/query \
--out-dir /path/to/evidence \
--execute
```
The runner requires Docker Compose, AWS CLI, curl, Warp through `run_object_batch_bench.sh`, Python, the four-node local cluster compose file, prebuilt candidate/baseline images, and an already reachable Prometheus/OTLP pipeline. It never builds, pulls, or starts observability images implicitly. Execution rejects reused Compose resources or a non-empty evidence directory and performs a Prometheus vector preflight before starting the cluster. The workload summary must contain only successful zero-error rows with parseable request rates; the rollback phase must read byte-identical object data written before the version replacement.
## Stage 2 — Remove the proto JSON fields
Only after Stage 1 has been stable fleet-wide for a full window with the counters still zero.
@@ -113,7 +154,7 @@ Only after Stage 1 has been stable fleet-wide for a full window with the counter
| Stage | Wire format broken? | Rollback |
| --- | --- | --- |
| 0 Observe | no | n/a (metrics only) |
| 1 msgpack-only send | no | unset either flag and restart |
| 1 msgpack-only send | no | unset the base request flag, or unset the selected fleet/peer gate, and restart |
| 2 remove fields | yes | redeploy the prior release; field numbers stay `reserved` |
## Related
+2
View File
@@ -207,3 +207,5 @@ Pending followers use indexed slab slots and a monotonic waiter identity. Detach
A source that disappears before native open fails its own waiter, while healthy requests in that batch still complete file and directory sync. Other file-sync errors retain the existing fail-fast batch behavior, preventing repeated I/O against a failing disk. Admission or native-dispatch failure also fails the batch. File sets and their existing resource units move together into one blocking dispatch, and result senders remain with the async worker so cancellation keeps its established notification contract.
With detailed PUT metrics enabled, a successful queue ownership transfer adds a `follower_timeout` role sample to `rustfs_s3_put_object_rename_fdatasync_group_wait_ms`. Evaluate the experiment against the same source, durability setting, topology, and workload. Include sync failures, follower publication before batch completion, cancellation, and peak resource usage in acceptance.
Grouped worker batches use `mode="grouped"` on `rustfs_s3_put_object_rename_fdatasync_batch_files`. The `parallel` mode is reserved for the independent per-file fanout path, so dashboards can distinguish serialized user-space grouping from bounded parallel dispatch.
@@ -898,42 +898,6 @@ done
The `aggregated.scanner` payload preserves the same scanner progress, checkpoint, pacing, source work, maintenance control, lifecycle expiry, and lifecycle transition fields used by the local scanner status, but only for the responding node; `by_host.*.scanner` keeps that node's host view. Compare the per-node artifacts externally to find old active paths, partial checkpoints, pacing pressure, source-level control pressure, or downstream queue admission problems across the deployment.
## Reading Cached Bucket Statistics
Admin usage reads cache the selected scanner baseline for the existing 30-second TTL, then verify every returned bucket against uncached storage-owned incarnation metadata. Identity reads use at most four concurrent operations and a shared two-second deadline after releasing snapshot/refresh locks. Each bucket read linearizes independently against metadata transactions; the response is not an atomic snapshot of the entire namespace. Delayed peer invalidation cannot authorize a predecessor's statistics for a recreated bucket.
Missing, failed, timed-out, nil, mismatched, or legacy absent identities leave the bucket value absent (unknown), rather than returning the predecessor's value or a confirmed zero. Legacy statistics become visible after a scanner snapshot supplies the current identity. Account-info and usage display apply the existing single-process memory overlay before the final identity filter; distributed local counters are not promoted. When a projection omits values or changes an overlay, totals are partial and `usage_snapshot_converged=false`, and per-tier/replication aggregates are unavailable. Verified cold bucket values remain visible. Such response-only projections cannot be persisted, used for quota admission, or acknowledge dirty work.
## Reading Scanner Work Observations
`GET /rustfs/admin/v3/scanner/status` exposes `metrics.scan_observation` for the responding node. Capture timestamped snapshots on every node, including the scanner leader, and compare their deltas over the same workload window. These process-lifetime counters reset on restart; a missing field on an older node means unobserved. They are loaded independently and may briefly disagree while operations finish. This telemetry does not change scan cadence, cached usage reads, quota authority, dirty acknowledgement, or maintenance requirements.
| Field | Meaning |
|---|---|
| `metadata_reads.attempts` | Disk metadata read futures started after scanner node admission. Invalid paths and operations cancelled before starting the read do not count. |
| `metadata_reads.succeeded`, `not_found`, `errors`, `cancelled` | Storage outcomes, including object/version disappearance and dropped read futures. Successful storage reads can still contain corrupt metadata; decoding and lifecycle failures do not become read errors. Cancellation observes the future, not proof that physical I/O stopped. |
| `metadata_reads.bytes` | Total returned buffer lengths for successful reads, including inline data and corrupt metadata. Failed reads contribute no known bytes; this is not physical disk traffic. |
| `metadata_reads.elapsed_nanos` | Sum of read-future elapsed time, including disk-permit or RPC waits, failures, and cancellation. Folder pacing, scanner node admission, decoding, and lifecycle work are outside this interval. Concurrent reads overlap, so the sum is not cycle wall time. |
| `cache_reuse.clean_prefix` | Clean root subtrees copied under an admitted prefix scope by the non-erasure scanner. |
| `cache_reuse.covered_namespace` | Namespace subtrees copied from validated checkpoint coverage. |
| `cache_reuse.sampled_namespace` | Cached namespace subtrees copied when the existing periodic sampling selected other work. |
| `cache_reuse.classified_namespace` | EC candidate namespace subtrees copied only after the current listing finished classifying them. Health/classification work still runs. |
| `cache_reuse.*.count`, `elapsed_nanos` | Completed subtree reuse decisions and time spent copying/attaching their cache entries. Counts are not objects skipped or a cache hit ratio; times exclude classification/listing and are not total cache maintenance cost. |
| `bucket_reuse.scoped_cold_bucket` | Whole cold buckets copied and attached to a successfully prepared scoped set cache. Count is copied buckets, not plans selected; later turn publication can still fail. Time covers copying/attachment only. |
| `bucket_reuse.current_root` | Existing current bucket roots successfully sent to the local collector after watch, namespace lease and publication epoch checks. Time covers channel send only; failure or cancellation contributes no completed reuse. Collector acceptance and durable publication remain separate. |
| `reuse_decisions.*.not_evaluated`, `blocked`, `eligible` | First conditions reached by production clean-prefix, checkpoint-coverage and namespace-sampling gates. Policy exclusions (`missing_scope`, `no_resume_sweep`, `deep_scan`, `forward_sweep`, `active_replication`) precede cache proof checks. `blocked` reports the first failed work/cache condition; later checks are not repeated for diagnostics. `eligible` does not certify a copy: an EC candidate can still require classification. |
| `erasure_clean_prefix_walks` | Clean root children that still require walking because EC scans perform independent object-health work. |
| `prefix_scope_resolution.evaluated`, `activated`, `blockers` | Proof evaluations retained by the final scope resolver, admitted proof gates, and every failed prerequisite. An admitted proof does not guarantee a reusable child exists; actual subtree copies appear in `cache_reuse`. Startup/required full scans, early scope rejection, and discarded peer/ACK resolutions do not count here. |
| `prefix_cycle_completion.evaluated`, `activated`, `blockers` | End-of-cycle proof evaluations for subsequent reuse. An activated completion proof does not mean this cycle reused prefixes or that its later publication/persistence succeeded. One evaluation can increment several blocker counters. |
| `scope_fallbacks` | Per-reason breakdown of the existing `usage_freshness.scope_fallback_total`; required full scans are excluded. |
| `authoritative_baseline_rejections`, `observed_baseline_rejections` | First failed validation checks for `baseline_unavailable` fallback cycles, preserving the authoritative rejection when the observed fallback also fails. Missing/undecodable authority never reaches observed validation; `authoritative_identity` means it could not anchor an observed fallback. |
`freshness` describes the responding node's completed-cycle age, not whether every bucket has incorporated new writes. Read `metrics.usage_freshness` for node-local live pending work, guardrail vetoes, durable success and publication state; read `metrics.usage_publication` for the confirmed snapshot's pending count and publication class/outcome, and `cycle_schedule` for effective retry intervals and consecutive superseded cycles. Missing or unknown publication data does not establish zero pending work. A follower or a single node's empty dirty map cannot establish cluster-wide convergence.
Whole cold bucket copies, current-root reuse and the four subtree categories have different prerequisites and are not interchangeable hit rates. EC clean prefixes retain health walks. Deep or required full work must run, but a completed cache matching that exact work digest may satisfy a retry of the same cycle. A normal workload need not exercise current-root reuse or every subtree category; use controlled tests for positive emission evidence and capture real gate reasons when a VM window has zero copies.
The existing `life_time_ops.read_metadata` and `last_minute.actions.read_metadata` now include finished and cancelled read attempts; the latter reports count, accumulated whole seconds, and returned bytes in its existing window. Use `metadata_reads.elapsed_nanos` for subsecond read costs. Use read-count and byte deltas to assess work growth, and elapsed-time deltas to assess read-future cost. Compare those with reused-subtree deltas, scope fallback reasons, cycle wall time, pacing waits, and source work before attributing freshness delays to metadata I/O or proposing a scan optimization.
## Reading Lifecycle Transition Status
`metrics.lifecycle_transition`:
+2 -1
View File
@@ -57,7 +57,8 @@ use tracing::{Span, debug, error, info, instrument, warn};
use tracing_opentelemetry::OpenTelemetrySpanExt;
#[derive(RustEmbed)]
#[folder = "$CARGO_MANIFEST_DIR/static"]
#[cfg_attr(test, folder = "$CARGO_MANIFEST_DIR/tests/fixtures/native-console")]
#[cfg_attr(not(test), folder = "$CARGO_MANIFEST_DIR/static")]
struct StaticFiles;
/// Static file handler
+3 -2
View File
@@ -20,7 +20,7 @@ use crate::admin::service::caller_identity::oidc_profile_fields;
use crate::admin::storage_api::bucket::versioning_sys::BucketVersioningSys;
use crate::admin::storage_api::contract::admin::StorageAdminApi;
use crate::admin::storage_api::contract::bucket::{BucketOperations, BucketOptions};
use crate::admin::storage_api::data_usage::load_admin_data_usage_with_memory_overlay;
use crate::admin::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached};
use crate::admin::storage_api::metadata_sys;
use crate::auth::get_condition_values;
use crate::server::ADMIN_PREFIX;
@@ -278,7 +278,8 @@ impl Operation for AccountInfoHandler {
// process-local absolute counters to cluster-wide bucket totals.
// This path never triggers a live full-version listing
// (rustfs/backlog#1306); freshness is owned by the scanner.
let data_usage_info = map_data_usage_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
let mut data_usage_info = map_data_usage_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
apply_bucket_usage_memory_overlay(&mut data_usage_info).await;
account_info.usage_snapshot_converged = data_usage_info.usage_snapshot_converged;
for bucket in buckets.iter() {
+8
View File
@@ -922,6 +922,14 @@ pub(crate) fn claims_route(method: &Method, uri: &Uri, headers: &HeaderMap) -> b
|| is_control_plane_route(method, uri, headers)
}
/// Console paths whose requests do not belong to an earlier extension route.
#[cfg(feature = "gateway-stack")]
pub(crate) fn claims_native_console_route(method: &Method, uri: &Uri) -> bool {
is_console_path(uri.path())
&& parse_object_lambda_request(method, uri).is_none()
&& s3_extension::route_key(method, uri).is_none()
}
fn is_public_health_path(path: &str) -> bool {
path == HEALTH_PREFIX || path == HEALTH_READY_PATH
}
+4 -6
View File
@@ -945,6 +945,10 @@ pub(crate) static ERR_TIER_RESERVED_NAME: AdminErrorRef = AdminErrorRef(|| &ecst
pub(crate) mod data_usage {
use std::sync::Arc;
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_data_usage_from_backend(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
@@ -956,12 +960,6 @@ pub(crate) mod data_usage {
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
}
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
}
}
pub(crate) mod access {
+22 -65
View File
@@ -21,7 +21,9 @@ use super::storage_api::admin_usecase::capacity::{
};
use super::storage_api::admin_usecase::contract::StorageAdminApi;
use super::storage_api::admin_usecase::contract::bucket::{BucketOperations as _, BucketOptions};
use super::storage_api::admin_usecase::data_usage::load_admin_data_usage_with_memory_overlay;
use super::storage_api::admin_usecase::data_usage::{
apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached,
};
use super::storage_api::admin_usecase::{ECStore, EndpointServerPools};
use crate::app::runtime_sources::{
AppContext, current_app_context, current_endpoints_handle, current_object_store_handle_for_context,
@@ -270,7 +272,7 @@ impl DefaultAdminUsecase {
info.buckets_usage.retain(|bucket, _| current_buckets.contains(bucket));
info.bucket_sizes.retain(|bucket, _| current_buckets.contains(bucket));
if info.buckets_usage.is_empty() && !current_buckets.is_empty() && !info.usage_snapshot_admin_projection {
if info.buckets_usage.is_empty() && !current_buckets.is_empty() {
*info = DataUsageInfo::default();
return;
}
@@ -285,14 +287,19 @@ impl DefaultAdminUsecase {
}
}
/// Keep already identity-bound display values without accepting a display
/// projection as a persistable or quota-authoritative snapshot.
fn present_identity_bound_data_usage_snapshot(
mut info: DataUsageInfo,
buckets: impl IntoIterator<Item = String>,
) -> DataUsageInfo {
let snapshot_measured =
info.usage_snapshot_admin_projection || info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
/// Overlay single-process counts on a persisted snapshot and narrow it
/// for display.
///
/// Whether the snapshot measured anything is decided before the overlay.
/// The overlay clears `usage_snapshot_complete` when it holds counts it
/// cannot reconcile with the snapshot, such as writes into a bucket that
/// has no verified baseline. That makes the totals unconverged, not the
/// scanned buckets unusable; treating it as structural incompleteness
/// blanked usage for every bucket for as long as one such bucket kept
/// receiving writes, because each write restarts its scanner fence.
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
apply_bucket_usage_memory_overlay(&mut info).await;
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
info
}
@@ -338,7 +345,7 @@ impl DefaultAdminUsecase {
/// This path never triggers a live full-version listing
/// (rustfs/backlog#1306); freshness is owned by the scanner.
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
let info = Self::map_data_usage_load_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
let buckets = store
.list_bucket(&BucketOptions {
cached: true,
@@ -347,7 +354,7 @@ impl DefaultAdminUsecase {
})
.await
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
let mut info = Self::present_identity_bound_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name));
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
@@ -972,52 +979,6 @@ mod tests {
assert_eq!(info.buckets_count, 0);
}
#[test]
fn identity_bound_partial_display_keeps_valid_buckets_and_unknown_state() {
use std::collections::HashMap;
let mut projected = DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
usage_snapshot_admin_projection: true,
usage_snapshot_partial: true,
usage_snapshot_converged: Some(false),
buckets_count: 1,
objects_total_count: 7,
objects_total_size: 70,
buckets_usage: HashMap::from([(
"cold".to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 70,
objects_count: 7,
..Default::default()
},
)]),
bucket_sizes: HashMap::from([("cold".to_string(), 70)]),
..Default::default()
};
projected = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
projected,
["cold".to_string(), "recreated".to_string()],
);
assert_eq!(projected.buckets_usage["cold"].objects_count, 7);
assert!(!projected.buckets_usage.contains_key("recreated"));
assert_eq!(projected.objects_total_count, 7);
assert_eq!(projected.usage_snapshot_converged, Some(false));
assert!(!projected.is_complete_bucket_usage_snapshot());
assert!(!projected.is_valid_partial_snapshot());
projected.buckets_usage.clear();
projected.bucket_sizes.clear();
projected.calculate_totals();
projected.buckets_count = 0;
let unknown = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(projected, ["recreated".to_string()]);
assert!(unknown.buckets_usage.is_empty());
assert!(unknown.usage_snapshot_admin_projection);
assert_eq!(unknown.usage_snapshot_converged, Some(false));
assert!(!unknown.is_complete_bucket_usage_snapshot());
assert_eq!(unknown.last_update, Some(std::time::SystemTime::UNIX_EPOCH));
}
/// A write into a bucket without a verified baseline leaves process-local
/// counts the overlay cannot reconcile, so it clears
/// `usage_snapshot_complete`. That must flag the response non-converged,
@@ -1048,15 +1009,11 @@ mod tests {
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
// Exercise the production overlay, followed by the same identity-bound
// presentation function used by the admin query.
crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut persisted).await;
persisted.usage_snapshot_admin_projection = true;
persisted.usage_snapshot_converged = Some(false);
let info = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
let info = DefaultAdminUsecase::present_data_usage_snapshot(
persisted,
[measured_bucket.to_string(), unverified_bucket.to_string()],
);
)
.await;
assert_eq!(
info.buckets_usage
+35 -9
View File
@@ -33,9 +33,9 @@ use super::storage_api::bucket_usecase::bucket::{
run_stale_multipart_upload_cleanup_once, validate_lifecycle_config, validate_transition_tier,
},
metadata::{
BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG,
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, BUCKET_REPLICATION_CONFIG, BUCKET_SSECONFIG, BUCKET_TARGETS_FILE,
BUCKET_VERSIONING_CONFIG,
BUCKET_ACCELERATE_CONFIG, BUCKET_CORS_CONFIG, BUCKET_LIFECYCLE_CONFIG, BUCKET_NOTIFICATION_CONFIG, BUCKET_POLICY_CONFIG,
BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, BUCKET_REPLICATION_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_SSECONFIG,
BUCKET_TARGETS_FILE, BUCKET_VERSIONING_CONFIG, BUCKET_WEBSITE_CONFIG,
},
metadata_sys,
policy_sys::PolicySys,
@@ -61,10 +61,7 @@ use super::storage_api::bucket_usecase::s3_api::bucket::{
ListObjectVersionsParams, ListObjectsV2Params, encode_list_objects_v2_continuation_token, parse_list_object_versions_params,
parse_list_objects_v2_params, rustfs_owner,
};
use super::storage_api::bucket_usecase::{
get_validated_store, process_lambda_configurations, process_queue_configurations, process_topic_configurations,
request_context, validate_list_object_unordered_with_delimiter,
};
use super::storage_api::bucket_usecase::{get_validated_store, request_context, validate_list_object_unordered_with_delimiter};
#[cfg(feature = "diagnostic-read-version-caller")]
use super::storage_api::diagnostic;
use crate::app::RequestEnvelope;
@@ -95,7 +92,6 @@ use rustfs_gateway_types::persistence::{
PersistedVersioningConfiguration,
};
use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedReplicationConfiguration};
use rustfs_io_metrics::record_s3_op;
use rustfs_madmin::{SITE_REPL_API_VERSION, SRBucketMeta};
use rustfs_policy::policy::{
action::{Action, S3Action},
@@ -128,6 +124,8 @@ mod core;
#[cfg(any(test, feature = "gateway-stack"))]
pub(crate) use self::core::requested_location_constraint;
mod cors;
#[cfg(test)]
mod delete_config_tests;
mod encryption;
#[cfg(test)]
mod guards;
@@ -138,6 +136,7 @@ pub(crate) mod list_objects;
mod listing;
mod logging;
mod notification;
pub(crate) use notification::NotificationDestination;
mod object_lock;
mod policy;
mod public_access_block;
@@ -275,6 +274,23 @@ pub(crate) async fn notify_table_bucket_metadata_reload(
notify_bucket_metadata_reload_with_context(bucket, operation, None, false, app_context).await;
}
#[cfg(test)]
type BucketMetadataReloadObserver = Box<dyn Fn(bool, &'static str, bool) + Send + Sync>;
#[cfg(test)]
tokio::task_local! {
static BUCKET_METADATA_RELOAD_PROBE: (String, BucketMetadataReloadObserver);
}
#[cfg(test)]
fn record_bucket_metadata_reload(bucket: &str, completed: bool, operation: &'static str, scanner_maintenance_change: bool) {
let _ = BUCKET_METADATA_RELOAD_PROBE.try_with(|(observed_bucket, record)| {
if observed_bucket == bucket {
record(completed, operation, scanner_maintenance_change);
}
});
}
async fn notify_bucket_metadata_reload_with_context(
bucket: String,
operation: &'static str,
@@ -282,6 +298,11 @@ async fn notify_bucket_metadata_reload_with_context(
scanner_maintenance_change: bool,
app_context: Option<Arc<AppContext>>,
) {
#[cfg(test)]
record_bucket_metadata_reload(&bucket, false, operation, scanner_maintenance_change);
#[cfg(test)]
let observed_bucket = bucket.clone();
record_local_scanner_maintenance_reload(&bucket, scanner_maintenance_change);
// Keep reload detached across request cancellation, but wait before a healthy peer can serve the previous config.
let (completed_tx, completed_rx) = tokio::sync::oneshot::channel();
@@ -303,7 +324,12 @@ async fn notify_bucket_metadata_reload_with_context(
}
let _ = completed_tx.send(());
});
let _ = completed_rx.await;
let completed = completed_rx.await;
#[cfg(test)]
if completed.is_ok() {
record_bucket_metadata_reload(&observed_bucket, true, operation, scanner_maintenance_change);
}
let _ = completed;
}
fn record_local_scanner_maintenance_reload(bucket: &str, scanner_maintenance_change: bool) {
+41 -1
View File
@@ -12,4 +12,44 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bucket accelerate configuration use-case operations (reserved for the next batch).
//! Bucket accelerate configuration shared by the S3 edges.
use super::BUCKET_ACCELERATE_CONFIG;
use super::{
ApiError, DefaultBucketUsecase, StorageError, bucket_config_mutation_incarnation, metadata_sys, serialize,
update_bucket_config_for_incarnation,
};
use crate::app::RequestEnvelope;
use rustfs_gateway_types::persistence::PersistedAccelerateConfiguration;
use rustfs_s3_types::S3Result;
impl DefaultBucketUsecase {
pub(crate) async fn execute_get_bucket_accelerate_configuration(
&self,
bucket: &str,
) -> S3Result<PersistedAccelerateConfiguration> {
self.ensure_config_bucket(bucket).await?;
match metadata_sys::get_accelerate_config(bucket).await {
Ok((config, _)) => Ok(config),
Err(StorageError::ConfigNotFound) => Ok(PersistedAccelerateConfiguration::default()),
Err(err) => Err(ApiError::from(err).into()),
}
}
pub(crate) async fn execute_put_bucket_accelerate_configuration(
&self,
envelope: RequestEnvelope,
bucket: &str,
configuration: impl FnOnce() -> S3Result<PersistedAccelerateConfiguration> + Send,
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
self.ensure_config_bucket(bucket).await?;
let config = configuration()?;
let data = serialize(&config).map_err(|err| rustfs_s3_types::s3_error!(MalformedXML, "{err}"))?;
update_bucket_config_for_incarnation(bucket, BUCKET_ACCELERATE_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
Ok(())
}
}
+19
View File
@@ -17,6 +17,25 @@
use super::*;
impl DefaultBucketUsecase {
pub(crate) async fn execute_put_bucket_acl(
&self,
envelope: RequestEnvelope,
bucket: &str,
has_xml_policy: bool,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutBucketAcl, StatusCode::OK, |_| async move {
self.ensure_config_bucket(bucket).await?;
if has_xml_policy {
return Err(rustfs_s3_types::s3_error!(
NotImplemented,
"ACL XML grants are not supported; use canned ACL headers or omit ACL"
));
}
Ok(())
})
.await
}
/// RustFS has one owner and no ACL storage: every bucket answers that owner with full control.
pub async fn execute_get_bucket_acl(
&self,
+12
View File
@@ -55,6 +55,18 @@ pub(crate) fn requested_location_constraint(input: &gateway_dto::CreateBucketInp
}
impl DefaultBucketUsecase {
/// Checks that the bucket exists before reading or writing a stored configuration.
pub(super) async fn ensure_config_bucket(&self, bucket: &str) -> rustfs_s3_types::S3Result<()> {
let Some(store) = self.object_store() else {
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
Ok(())
}
fn create_bucket_exists_result(&self) -> rustfs_s3_types::S3Result<gateway_dto::CreateBucketOutput> {
let region = self
.global_region()
+17 -21
View File
@@ -15,8 +15,6 @@
//! Bucket CORS configuration use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::GetBucketCorsInput;
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Result};
use rustfs_gateway_types::cors_tagging::parse_cors;
use rustfs_gateway_types::persistence::serialize_cors_dto;
@@ -64,26 +62,30 @@ impl DefaultBucketUsecase {
.await
}
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_get_bucket_cors(&self, req: S3Request<GetBucketCorsInput>) -> S3Result<PersistedCorsConfiguration> {
let GetBucketCorsInput { bucket, .. } = req.input;
/// Returns the stored bucket CORS configuration after checking the bucket exists.
///
/// # Errors
///
/// Returns `InternalError` if the store is uninitialized, `NoSuchCORSConfiguration`
/// if the configuration is absent, or the storage error from either lookup.
#[instrument(level = "debug", skip(self, bucket))]
pub async fn execute_get_bucket_cors(&self, bucket: &str) -> rustfs_s3_types::S3Result<PersistedCorsConfiguration> {
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let cors_configuration = match metadata_sys::get_cors_config(&bucket).await {
let cors_configuration = match metadata_sys::get_cors_config(bucket).await {
Ok((config, _)) => config,
Err(err) => {
if err == StorageError::ConfigNotFound {
return Err(S3Error::with_message(
S3ErrorCode::NoSuchCORSConfiguration,
"The CORS configuration does not exist".to_string(),
return Err(rustfs_s3_types::s3_error!(
NoSuchCORSConfiguration,
"The CORS configuration does not exist"
));
}
warn!(
@@ -154,7 +156,7 @@ impl DefaultBucketUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::{build_request, gateway_bucket, owned_envelope};
use crate::app::bucket_usecase::test_support::{gateway_bucket, owned_envelope};
use http::Method;
#[tokio::test]
@@ -174,16 +176,10 @@ mod tests {
#[tokio::test]
async fn execute_get_bucket_cors_returns_internal_error_when_store_uninitialized() {
let input = GetBucketCorsInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_get_bucket_cors(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let err = usecase.execute_get_bucket_cors("test-bucket").await.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
}
#[tokio::test]
@@ -0,0 +1,350 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bucket configuration mutation tests over the shared use case and durable metadata.
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::owned_parity::authorize_bucket_config_mutation;
use crate::site_replication::hooks::BUCKET_META_HOOK_CALLS;
use http::Method;
use rustfs_credentials::Credentials;
use rustfs_s3_types::{S3ErrorCode, S3Result};
use serial_test::serial;
use std::{cell::RefCell, sync::Mutex};
#[derive(Clone, Copy)]
enum Config {
Encryption,
Policy,
Tagging,
}
impl Config {
async fn seed(self, bucket: &str) {
match self {
Self::Encryption => {
metadata_sys::update(
bucket,
BUCKET_SSECONFIG,
b"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault>\
<SSEAlgorithm>AES256</SSEAlgorithm></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>"
.to_vec(),
)
.await
}
Self::Policy => {
metadata_sys::update(bucket, BUCKET_POLICY_CONFIG, br#"{"Version":"2012-10-17","Statement":[]}"#.to_vec()).await
}
Self::Tagging => {
let tags = rustfs_gateway_types::cors_tagging::parse_tagging(
b"<Tagging><TagSet><Tag><Key>team</Key><Value>storage</Value></Tag></TagSet></Tagging>",
)
.expect("valid tagging fixture");
metadata_sys::update_s3_tagging_config(bucket, Some(tags), None).await
}
}
.expect("seed bucket configuration");
}
async fn stored(self, bucket: &str) -> Vec<u8> {
let metadata = metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("read bucket metadata from disk");
match self {
Self::Encryption => metadata.encryption_config_xml,
Self::Policy => metadata.policy_config_json,
Self::Tagging => metadata.tagging_config_xml,
}
}
async fn put(self, usecase: &DefaultBucketUsecase, envelope: RequestEnvelope, bucket: &str) -> S3Result<()> {
match self {
Self::Encryption => {
let config = rustfs_gateway_types::persistence::parse_bucket_encryption(ENCRYPTION_WRITE).unwrap();
usecase.execute_put_bucket_encryption(envelope, bucket, Ok(config)).await
}
Self::Policy => usecase.execute_put_bucket_policy(envelope, bucket, POLICY_WRITE).await,
Self::Tagging => {
let tags = rustfs_gateway_types::cors_tagging::parse_tagging(TAGGING_WRITE).unwrap();
usecase.execute_put_bucket_tagging(envelope, bucket, tags).await
}
}
}
async fn execute(self, usecase: &DefaultBucketUsecase, envelope: RequestEnvelope, bucket: &str) -> S3Result<()> {
match self {
Self::Encryption => usecase.execute_delete_bucket_encryption(envelope, bucket).await,
Self::Policy => usecase.execute_delete_bucket_policy(envelope, bucket).await,
Self::Tagging => usecase.execute_delete_bucket_tagging(envelope, bucket).await,
}
}
}
const ENCRYPTION_WRITE: &[u8] = b"<ServerSideEncryptionConfiguration><Rule><ApplyServerSideEncryptionByDefault><SSEAlgorithm>aws:kms</SSEAlgorithm><KMSMasterKeyID>fixture-key</KMSMasterKeyID></ApplyServerSideEncryptionByDefault></Rule></ServerSideEncryptionConfiguration>";
const POLICY_WRITE: &str = "{ \"Version\": \"2012-10-17\", \"Statement\": [] } ";
const TAGGING_WRITE: &[u8] = b"<Tagging><TagSet><Tag><Key>team</Key><Value>gateway</Value></Tag></TagSet></Tagging>";
async fn check_mutation(config: Config, stale: bool, write: bool) {
let (subresource, action, operation, reload, replicated, audit_status) = match (config, write) {
(Config::Encryption, false) => (
"encryption",
S3Action::PutBucketEncryptionAction,
S3Operation::DeleteBucketEncryption,
"delete bucket encryption",
"sse-config",
204,
),
(Config::Policy, false) => (
"policy",
S3Action::DeleteBucketPolicyAction,
S3Operation::DeleteBucketPolicy,
"delete bucket policy",
"policy",
200,
),
(Config::Tagging, false) => (
"tagging",
S3Action::PutBucketTaggingAction,
S3Operation::DeleteBucketTagging,
"delete bucket tagging",
"tags",
200,
),
(Config::Encryption, true) => (
"encryption",
S3Action::PutBucketEncryptionAction,
S3Operation::PutBucketEncryption,
"put bucket encryption",
"sse-config",
200,
),
(Config::Policy, true) => (
"policy",
S3Action::PutBucketPolicyAction,
S3Operation::PutBucketPolicy,
"put bucket policy",
"policy",
200,
),
(Config::Tagging, true) => (
"tagging",
S3Action::PutBucketTaggingAction,
S3Operation::PutBucketTagging,
"put bucket tagging",
"tags",
200,
),
};
let context = shared_gating_ambient().await;
let store = context.object_store();
let bucket = format!("config-mutation-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(&context)));
let mut request = build_request((), if write { Method::PUT } else { Method::DELETE });
request.uri = format!("/{bucket}?{subresource}").parse().expect("request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "config-delete-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
request.extensions.insert(request_context::RequestContext {
request_id: "bucket-config-delete".to_owned(),
x_amz_request_id: "bucket-config-delete".to_owned(),
trace_id: None,
span_id: None,
start_time: std::time::Instant::now(),
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(&server, &mut envelope, &bucket, Action::S3Action(action))
.await
.expect("capture authorized bucket incarnation");
if stale {
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete authorized bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
}
config.seed(&bucket).await;
let seeded = config.stored(&bucket).await;
assert!(!seeded.is_empty(), "durable configuration fixture");
let before = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let maintenance_before = rustfs_scanner::scanner_maintenance_generation();
let audits = MutationCompletionProbe::install(&bucket);
let reloads = Arc::new(Mutex::new(Vec::new()));
let observe = Arc::clone(&reloads);
let usecase = DefaultBucketUsecase::with_context(Some(context));
let (result, hooks) = BUCKET_META_HOOK_CALLS
.scope(RefCell::default(), async {
let result = BUCKET_METADATA_RELOAD_PROBE
.scope(
(
bucket.clone(),
Box::new(move |completed, label, maintenance| {
let replication_calls = BUCKET_META_HOOK_CALLS.with(|calls| calls.borrow().len());
observe
.lock()
.expect("reload entries")
.push((completed, label, maintenance, replication_calls));
}),
),
async {
if write {
config.put(&usecase, envelope, &bucket).await
} else {
config.execute(&usecase, envelope, &bucket).await
}
},
)
.await;
(result, BUCKET_META_HOOK_CALLS.with(RefCell::take))
})
.await;
let stored = config.stored(&bucket).await;
let after = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let reloads = reloads.lock().expect("reload entries");
let entries = audits.audits.lock().expect("audit entries");
assert!(audits.events.lock().expect("event entries").is_empty());
let audit = entries
.iter()
.map(|entry| (entry.api.name.as_deref(), entry.api.status_code))
.collect::<Vec<_>>();
if stale {
assert_eq!(
(
result.err().map(|err| err.code().clone()),
stored,
reloads.len(),
hooks.len(),
after,
audit
),
(
Some(S3ErrorCode::NoSuchBucket),
seeded,
0,
0,
before,
vec![(Some(operation.as_str()), Some(404))]
),
"{operation:?} must refuse the stale authorization and preserve replacement bytes"
);
} else {
result.expect("authorized mutation");
if write {
match config {
Config::Encryption => {
let actual = rustfs_gateway_types::persistence::parse_bucket_encryption(&stored).unwrap();
let expected = rustfs_gateway_types::persistence::parse_bucket_encryption(ENCRYPTION_WRITE).unwrap();
assert_eq!(actual, expected);
}
Config::Policy => assert_eq!(stored, POLICY_WRITE.as_bytes()),
Config::Tagging => {
let actual = rustfs_gateway_types::cors_tagging::parse_tagging(&stored).unwrap();
let expected = rustfs_gateway_types::cors_tagging::parse_tagging(TAGGING_WRITE).unwrap();
assert_eq!(actual, expected);
}
}
} else {
assert!(stored.is_empty(), "{operation:?} must remove the durable configuration");
}
assert_eq!(*reloads, [(false, reload, false, 0), (true, reload, false, 0)]);
let [hook] = hooks.as_slice() else { panic!("{operation:?}: expected one replication call") };
assert_eq!((&hook.bucket, hook.item_type.as_str()), (&bucket, replicated));
assert_eq!(hook.scanner_maintenance_generation, maintenance_before);
assert_eq!(rustfs_scanner::scanner_maintenance_generation(), maintenance_before);
if matches!(config, Config::Tagging) {
assert!(after > hook.scanner_generation);
} else {
assert_eq!(after, hook.scanner_generation);
}
assert_eq!(audit, [(Some(operation.as_str()), Some(audit_status))]);
assert_eq!(entries[0].request_id.as_deref(), Some("bucket-config-delete"));
}
}
#[test]
#[serial]
fn encryption_delete_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("delete-encryption", || async {
check_mutation(Config::Encryption, false, false).await;
check_mutation(Config::Encryption, true, false).await;
});
}
#[test]
#[serial]
fn policy_delete_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("delete-policy", || async {
check_mutation(Config::Policy, false, false).await;
check_mutation(Config::Policy, true, false).await;
});
}
#[test]
#[serial]
fn tagging_delete_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("delete-tagging", || async {
check_mutation(Config::Tagging, false, false).await;
check_mutation(Config::Tagging, true, false).await;
});
}
#[test]
#[serial]
fn encryption_put_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("put-encryption", || async {
check_mutation(Config::Encryption, false, true).await;
check_mutation(Config::Encryption, true, true).await;
});
}
#[test]
#[serial]
fn policy_put_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("put-policy", || async {
check_mutation(Config::Policy, false, true).await;
check_mutation(Config::Policy, true, true).await;
});
}
#[test]
#[serial]
fn tagging_put_fences_recreated_bucket_and_orders_callbacks() {
run_large_stack_test("put-tagging", || async {
check_mutation(Config::Tagging, false, true).await;
check_mutation(Config::Tagging, true, true).await;
});
}
+121 -104
View File
@@ -15,17 +15,11 @@
//! Bucket default-encryption use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{
DeleteBucketEncryptionInput, DeleteBucketEncryptionOutput, GetBucketEncryptionInput, PutBucketEncryptionInput,
PutBucketEncryptionOutput, ServerSideEncryption,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
/// Refuse a default-encryption configuration the write path could not honour
/// as written. `bucket_default_write_sse` falls back to AES256 for any
/// algorithm it does not know, so storing one would make GetBucketEncryption
/// advertise a scheme no object is encrypted under. s3s parses `SSEAlgorithm`
/// as an open string, so the schema check has to happen here.
/// as written. `bucket_default_write_sse` falls back to AES256 for an unknown algorithm,
/// which would make GetBucketEncryption advertise a scheme the write path cannot use.
fn validate_bucket_encryption_configuration(config: &PersistedBucketEncryptionConfiguration) -> S3Result<()> {
if config.rules.is_empty() {
return Err(S3Error::with_message(
@@ -42,14 +36,14 @@ fn validate_bucket_encryption_configuration(config: &PersistedBucketEncryptionCo
};
let names_kms_key = by_default.kms_master_key_id.as_deref().is_some_and(|id| !id.is_empty());
match by_default.sse_algorithm.as_str() {
ServerSideEncryption::AWS_KMS => {}
ServerSideEncryption::AES256 if names_kms_key => {
"aws:kms" => {}
"AES256" if names_kms_key => {
return Err(S3Error::with_message(
S3ErrorCode::InvalidArgument,
"KMSMasterKeyID can only be specified when SSEAlgorithm is aws:kms".to_string(),
));
}
ServerSideEncryption::AES256 => {}
"AES256" => {}
other => {
return Err(S3Error::with_message(
S3ErrorCode::MalformedXML,
@@ -62,57 +56,72 @@ fn validate_bucket_encryption_configuration(config: &PersistedBucketEncryptionCo
}
impl DefaultBucketUsecase {
pub async fn execute_delete_bucket_encryption(
pub(crate) async fn execute_delete_bucket_encryption(
&self,
req: S3Request<DeleteBucketEncryptionInput>,
) -> S3Result<S3Response<DeleteBucketEncryptionOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeleteBucketEncryptionInput { bucket, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::DeleteBucketEncryption,
StatusCode::NO_CONTENT,
|envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(&bucket, BUCKET_SSECONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(bucket, BUCKET_SSECONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "delete bucket encryption", request_context, false).await;
notify_bucket_metadata_reload(bucket.to_owned(), "delete bucket encryption", request_context, false).await;
let item = sr_bucket_meta_item(bucket.clone(), "sse-config");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket encryption delete hook failed");
}
let item = sr_bucket_meta_item(bucket.to_owned(), "sse-config");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket encryption delete hook failed");
}
Ok(S3Response::with_status(DeleteBucketEncryptionOutput::default(), StatusCode::NO_CONTENT))
Ok(())
},
)
.await
}
/// Returns the stored bucket encryption configuration after checking the bucket exists.
///
/// # Errors
///
/// Returns `InternalError` if the store is uninitialized, `ServerSideEncryptionConfigurationNotFoundError`
/// if the configuration is absent, or the storage error from either lookup.
pub async fn execute_get_bucket_encryption(
&self,
req: S3Request<GetBucketEncryptionInput>,
) -> S3Result<Option<PersistedBucketEncryptionConfiguration>> {
let GetBucketEncryptionInput { bucket, .. } = req.input;
bucket: &str,
) -> rustfs_s3_types::S3Result<PersistedBucketEncryptionConfiguration> {
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let server_side_encryption_configuration = match metadata_sys::get_sse_config(&bucket).await {
Ok((cfg, _)) => Some(cfg),
let server_side_encryption_configuration = match metadata_sys::get_sse_config(bucket).await {
Ok((cfg, _)) => cfg,
Err(err) => {
if err == StorageError::ConfigNotFound {
return Err(s3_error!(ServerSideEncryptionConfigurationNotFoundError));
return Err(rustfs_s3_types::s3_error!(
ServerSideEncryptionConfigurationNotFoundError,
"The server-side encryption configuration was not found."
));
}
warn!(
component = LOG_COMPONENT_APP,
@@ -129,64 +138,65 @@ impl DefaultBucketUsecase {
Ok(server_side_encryption_configuration)
}
pub async fn execute_put_bucket_encryption(
pub(crate) async fn execute_put_bucket_encryption(
&self,
req: S3Request<PutBucketEncryptionInput>,
prepared: S3Result<PersistedBucketEncryptionConfiguration>,
) -> S3Result<S3Response<PutBucketEncryptionOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let PutBucketEncryptionInput { bucket, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
prepared: rustfs_s3_types::S3Result<PersistedBucketEncryptionConfiguration>,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutBucketEncryption, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let mut server_side_encryption_configuration = prepared?;
validate_bucket_encryption_configuration(&server_side_encryption_configuration)?;
let mut server_side_encryption_configuration = prepared?;
validate_bucket_encryption_configuration(&server_side_encryption_configuration)?;
// When SSE-KMS is set without a specific key ID, populate the default
// KMS key so that GetBucketEncryption responses include it. Clients like
// mc rely on the presence of KMSMasterKeyID to distinguish SSE-KMS from
// SSE-S3 in their display logic.
if let Some(rule) = server_side_encryption_configuration.rules.first_mut()
&& let Some(ref mut by_default) = rule.apply_server_side_encryption_by_default
&& by_default.sse_algorithm.as_str() == ServerSideEncryption::AWS_KMS
&& by_default.kms_master_key_id.as_deref().is_none_or(str::is_empty)
{
let service = current_encryption_service()
// When SSE-KMS is set without a specific key ID, populate the default
// KMS key so that GetBucketEncryption responses include it. Clients like
// mc rely on the presence of KMSMasterKeyID to distinguish SSE-KMS from
// SSE-S3 in their display logic.
if let Some(rule) = server_side_encryption_configuration.rules.first_mut()
&& let Some(ref mut by_default) = rule.apply_server_side_encryption_by_default
&& by_default.sse_algorithm.as_str() == "aws:kms"
&& by_default.kms_master_key_id.as_deref().is_none_or(str::is_empty)
{
let service = current_encryption_service()
.await
.ok_or_else(|| s3_error!(InternalError, "KMS service not initialized"))?;
let default_key = service
.get_default_key_id()
.filter(|key| !key.is_empty())
.ok_or_else(|| s3_error!(InternalError, "KMS default key not configured"))?;
by_default.kms_master_key_id = Some(default_key.clone());
}
let Some(store) = self.object_store() else {
return Err(s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "KMS service not initialized".to_string()))?;
let default_key = service
.get_default_key_id()
.filter(|key| !key.is_empty())
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "KMS default key not configured".to_string()))?;
by_default.kms_master_key_id = Some(default_key.clone());
}
.map_err(ApiError::from)?;
info!("sse_config {:?}", &server_side_encryption_configuration);
let data = serialize_config(&server_side_encryption_configuration)?;
update_bucket_config_for_incarnation(bucket, BUCKET_SSECONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
notify_bucket_metadata_reload(bucket.to_owned(), "put bucket encryption", request_context, false).await;
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let data = serialize_config(&server_side_encryption_configuration)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_SSECONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "put bucket encryption", request_context, false).await;
let mut item = sr_bucket_meta_item(bucket.clone(), "sse-config");
item.sse_config = Some(
serialize_config(&server_side_encryption_configuration)
.and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?,
);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket encryption hook failed");
}
Ok(S3Response::new(PutBucketEncryptionOutput::default()))
let mut item = sr_bucket_meta_item(bucket.to_owned(), "sse-config");
item.sse_config = Some(
serialize_config(&server_side_encryption_configuration)
.and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?,
);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket encryption hook failed");
}
Ok(())
})
.await
}
}
@@ -194,7 +204,7 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::storage_api::s3::dto;
use crate::app::storage_api::s3::dto::{self, PutBucketEncryptionInput, ServerSideEncryption};
use http::Method;
use rustfs_gateway_types::persistence::{
PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
@@ -257,16 +267,10 @@ mod tests {
#[tokio::test]
async fn execute_get_bucket_encryption_returns_internal_error_when_store_uninitialized() {
let input = GetBucketEncryptionInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_get_bucket_encryption(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let err = usecase.execute_get_bucket_encryption("test-bucket").await.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
}
#[tokio::test]
@@ -292,7 +296,20 @@ mod tests {
let prepared = crate::storage::persisted_from_s3s(&req.input.server_side_encryption_configuration);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_put_bucket_encryption(req, prepared).await.unwrap_err();
let err = usecase
.execute_put_bucket_encryption(RequestEnvelope::from_s3s(&req), &req.input.bucket, prepared.map_err(Into::into))
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let err = usecase
.execute_put_bucket_encryption(
RequestEnvelope::from_s3s(&req),
&req.input.bucket,
Ok(PersistedBucketEncryptionConfiguration { rules: vec![] }),
)
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::MalformedXML);
}
}
+221 -25
View File
@@ -15,8 +15,7 @@
//! Bucket lifecycle configuration use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{DeleteBucketLifecycleInput, DeleteBucketLifecycleOutput};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, s3_error};
use rustfs_gateway_types::persistence::{PersistedLifecycleRule, parse_lifecycle_dto};
const ERR_LIFECYCLE_RULE_STATUS: &str = "Rule status must be either Enabled or Disabled";
@@ -128,36 +127,44 @@ fn lifecycle_has_abort_multipart_rules(config: &PersistedLifecycleConfiguration)
}
impl DefaultBucketUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_delete_bucket_lifecycle(
#[instrument(level = "debug", skip(self, envelope, bucket))]
pub(crate) async fn execute_delete_bucket_lifecycle(
&self,
req: S3Request<DeleteBucketLifecycleInput>,
) -> S3Result<S3Response<DeleteBucketLifecycleOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeleteBucketLifecycleInput { bucket, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
) -> rustfs_s3_types::S3Result<gateway_dto::DeleteBucketLifecycleOutput> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::DeleteBucketLifecycle,
StatusCode::OK,
|envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(&bucket, BUCKET_LIFECYCLE_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(bucket, BUCKET_LIFECYCLE_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "delete bucket lifecycle", request_context, true).await;
notify_bucket_metadata_reload(bucket.to_owned(), "delete bucket lifecycle", request_context, true).await;
let item = sr_bucket_meta_item(bucket.clone(), "lc-config");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket lifecycle delete hook failed");
}
let item = sr_bucket_meta_item(bucket.to_owned(), "lc-config");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket lifecycle delete hook failed");
}
Ok(S3Response::new(DeleteBucketLifecycleOutput::default()))
Ok(gateway_dto::DeleteBucketLifecycleOutput::default())
},
)
.await
}
/// The stored lifecycle configuration as the gateway's answer holds it.
@@ -353,12 +360,201 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::owned_parity::{ReqInfo, authorize_bucket_config_mutation};
use crate::app::storage_api::s3::dto;
use crate::app::storage_api::s3::dto::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, NoncurrentVersionTransition,
PutBucketLifecycleConfigurationInput, Transition, TransitionStorageClass,
};
use crate::site_replication::hooks::{BUCKET_META_HOOK_CALLS, BucketMetaHookCall};
use http::Method;
use rustfs_credentials::Credentials;
use rustfs_policy::policy::action::{Action, S3Action};
use serial_test::serial;
use std::cell::RefCell;
use std::sync::Arc;
const DELETE_LIFECYCLE_XML: &[u8] = b"<LifecycleConfiguration><Rule><ID>kept</ID><Status>Disabled</Status>\
<Filter/><Expiration><Days>1</Days></Expiration></Rule></LifecycleConfiguration>";
const DELETE_REQUEST_ID: &str = "lifecycle-delete-context";
fn delete_envelope(context: &Arc<AppContext>, bucket: &str) -> (Arc<ServerContextSlot>, RequestEnvelope) {
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(context)));
let mut request = build_request((), Method::DELETE);
request.uri = format!("/{bucket}?lifecycle").parse().expect("bucket request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "lifecycle-delete-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
request.extensions.insert(request_context::RequestContext {
request_id: DELETE_REQUEST_ID.to_owned(),
x_amz_request_id: DELETE_REQUEST_ID.to_owned(),
trace_id: None,
span_id: None,
start_time: std::time::Instant::now(),
});
(server, RequestEnvelope::from_s3s(&request))
}
async fn authorized_lifecycle_envelope(context: &Arc<AppContext>, bucket: &str) -> RequestEnvelope {
let (server, mut envelope) = delete_envelope(context, bucket);
authorize_bucket_config_mutation(&server, &mut envelope, bucket, Action::S3Action(S3Action::PutBucketLifecycleAction))
.await
.expect("bucket owner authorization installs the incarnation snapshot");
envelope
}
async fn seed_lifecycle(bucket: &str) {
metadata_sys::update(bucket, BUCKET_LIFECYCLE_CONFIG, DELETE_LIFECYCLE_XML.to_vec())
.await
.expect("store lifecycle configuration");
}
async fn persisted_lifecycle(bucket: &str) -> Vec<u8> {
metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("persisted bucket metadata")
.lifecycle_config_xml
}
async fn observed_delete(
usecase: &DefaultBucketUsecase,
envelope: RequestEnvelope,
bucket: &str,
) -> (
rustfs_s3_types::S3Result<gateway_dto::DeleteBucketLifecycleOutput>,
Vec<BucketMetaHookCall>,
) {
BUCKET_META_HOOK_CALLS
.scope(RefCell::default(), async move {
let result = usecase.execute_delete_bucket_lifecycle(envelope, bucket).await;
(result, BUCKET_META_HOOK_CALLS.with(RefCell::take))
})
.await
}
#[test]
#[serial]
fn delete_lifecycle_rejects_a_stale_bucket_incarnation() {
run_large_stack_test("delete-lifecycle-stale", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let bucket = format!("lifecycle-delete-stale-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create original bucket");
let envelope = authorized_lifecycle_envelope(&ambient, &bucket).await;
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
seed_lifecycle(&bucket).await;
let before = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let usecase = DefaultBucketUsecase::with_context(Some(ambient));
let (result, calls) = observed_delete(&usecase, envelope, &bucket).await;
let error = result.expect_err("a stale bucket incarnation cannot delete replacement configuration");
assert_eq!(error.code(), &rustfs_s3_types::S3ErrorCode::NoSuchBucket);
assert_eq!(persisted_lifecycle(&bucket).await, DELETE_LIFECYCLE_XML);
assert!(calls.is_empty());
assert_eq!(rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket), before);
});
}
#[test]
#[serial]
fn delete_lifecycle_rejects_misbound_or_missing_bucket_snapshots() {
run_large_stack_test("delete-lifecycle-unbound", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let authorized = format!("lifecycle-delete-authorized-{}", uuid::Uuid::new_v4().simple());
let target = format!("lifecycle-delete-target-{}", uuid::Uuid::new_v4().simple());
for bucket in [&authorized, &target] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
}
seed_lifecycle(&target).await;
let misbound = authorized_lifecycle_envelope(&ambient, &authorized).await;
let (_, missing) = delete_envelope(&ambient, &target);
let usecase = DefaultBucketUsecase::with_context(Some(ambient));
for (envelope, message) in [
(misbound, "bucket config mutation snapshot does not match request bucket"),
(missing, "bucket config mutation snapshot is missing"),
] {
let before = rustfs_scanner::scanner_dirty_usage_bucket_generation(&target);
let (result, calls) = observed_delete(&usecase, envelope, &target).await;
let error = result.expect_err("an unbound snapshot cannot delete lifecycle configuration");
assert_eq!(error.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
assert_eq!(error.message(), Some(message));
assert_eq!(persisted_lifecycle(&target).await, DELETE_LIFECYCLE_XML);
assert!(calls.is_empty());
assert_eq!(rustfs_scanner::scanner_dirty_usage_bucket_generation(&target), before);
}
});
}
#[test]
#[serial]
fn delete_lifecycle_reloads_scanner_and_audits_success_once() {
run_large_stack_test("delete-lifecycle-completion", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let bucket = format!("lifecycle-delete-audit-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
seed_lifecycle(&bucket).await;
let envelope = authorized_lifecycle_envelope(&ambient, &bucket).await;
let before = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let maintenance_before = rustfs_scanner::scanner_maintenance_generation();
let audit = MutationCompletionProbe::install(&bucket);
let usecase = DefaultBucketUsecase::with_context(Some(ambient));
let (result, calls) = observed_delete(&usecase, envelope, &bucket).await;
result.expect("delete lifecycle configuration");
assert!(persisted_lifecycle(&bucket).await.is_empty());
let [call] = calls.as_slice() else { panic!("expected one site replication hook call") };
assert_eq!(call.bucket, bucket);
assert_eq!(call.item_type, "lc-config");
// Persistence and the local lifecycle reload each advance the maintenance generation.
assert_eq!(
call.scanner_maintenance_generation,
maintenance_before.saturating_add(2),
"lifecycle reload advances maintenance before the site hook"
);
assert_eq!(rustfs_scanner::scanner_maintenance_generation(), call.scanner_maintenance_generation);
assert!(call.scanner_generation > before, "scanner reload precedes site replication");
assert_eq!(rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket), call.scanner_generation);
let audits = audit.audits.lock().expect("audit entries");
assert_eq!(audits.len(), 1, "one audit entry");
assert_eq!(audits[0].request_id.as_deref(), Some(DELETE_REQUEST_ID));
assert_eq!(audits[0].api.status_code, Some(200), "audit status uses 200");
});
}
#[tokio::test]
async fn execute_get_bucket_lifecycle_configuration_returns_internal_error_when_store_uninitialized() {
+138 -1
View File
@@ -12,4 +12,141 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bucket logging configuration use-case operations (reserved for the next batch).
//! Bucket logging configuration shared by the S3 edges.
use super::{
ApiError, DefaultBucketUsecase, S3Operation, StatusCode, StorageError, bucket_config_mutation_incarnation, metadata_sys,
update_bucket_config_for_incarnation,
};
use crate::app::RequestEnvelope;
use crate::app::storage_api::bucket_usecase::bucket::metadata::BUCKET_LOGGING_CONFIG;
use rustfs_gateway_types::persistence::PersistedBucketLoggingStatus;
use rustfs_s3_types::S3Result;
impl DefaultBucketUsecase {
pub(crate) async fn execute_put_bucket_logging(
&self,
envelope: RequestEnvelope,
bucket: &str,
serialize_configuration: impl FnOnce() -> S3Result<Vec<u8>> + Send,
) -> S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutBucketLogging, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
self.ensure_config_bucket(bucket).await?;
let data = serialize_configuration()?;
update_bucket_config_for_incarnation(bucket, BUCKET_LOGGING_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
Ok(())
})
.await
}
pub(crate) async fn execute_get_bucket_logging(&self, bucket: &str) -> S3Result<PersistedBucketLoggingStatus> {
self.ensure_config_bucket(bucket).await?;
match metadata_sys::get_logging_config(bucket).await {
Ok((config, _)) => Ok(config),
Err(StorageError::ConfigNotFound) => Ok(PersistedBucketLoggingStatus::default()),
Err(err) => Err(ApiError::from(err).into()),
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::{BucketOperations, DeleteBucketOptions, MakeBucketOptions};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::owned_parity::{ReqInfo, authorize_bucket_config_mutation};
use http::Method;
use rustfs_credentials::Credentials;
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_s3_types::S3ErrorCode;
use serial_test::serial;
use std::sync::Arc;
#[test]
#[serial]
fn put_bucket_logging_checks_bucket_before_serializing() {
run_large_stack_test("put-bucket-logging-lookup", || async {
let context = shared_gating_ambient().await;
let bucket = format!("logging-missing-{}", uuid::Uuid::new_v4().simple());
let usecase = DefaultBucketUsecase::with_context(Some(context));
let request = build_request((), Method::PUT);
let envelope = RequestEnvelope::from_s3s(&request);
let error = usecase
.execute_put_bucket_logging(envelope, &bucket, || {
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid logging configuration"))
})
.await
.unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::NoSuchBucket);
});
}
#[test]
#[serial]
fn put_bucket_logging_preserves_replacement_bytes_after_stale_authorization() {
run_large_stack_test("put-bucket-logging-incarnation", || async {
const ORIGINAL: &[u8] = b"<BucketLoggingStatus></BucketLoggingStatus>";
const UPDATE: &[u8] = b"<BucketLoggingStatus><LoggingEnabled><TargetBucket>other-bucket</TargetBucket><TargetPrefix>logs/</TargetPrefix></LoggingEnabled></BucketLoggingStatus>";
let context = shared_gating_ambient().await;
let store = context.object_store();
let bucket = format!("logging-stale-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create original bucket");
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(&context)));
let mut request = build_request((), Method::PUT);
request.uri = format!("/{bucket}?logging").parse().expect("logging request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "logging-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(&server, &mut envelope, &bucket, Action::S3Action(S3Action::PutBucketLoggingAction))
.await
.expect("snapshot the original bucket incarnation");
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create replacement bucket");
metadata_sys::update(&bucket, BUCKET_LOGGING_CONFIG, ORIGINAL.to_vec())
.await
.expect("seed replacement logging config");
let usecase = DefaultBucketUsecase::with_context(Some(context));
let result = usecase
.execute_put_bucket_logging(envelope, &bucket, || Ok(UPDATE.to_vec()))
.await;
let stored = metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(&bucket)
.await
.expect("replacement metadata")
.logging_config_xml;
assert_eq!(
(result.err().map(|err| err.code().clone()), stored.as_slice()),
(Some(S3ErrorCode::NoSuchBucket), ORIGINAL),
"the stale authorization must leave the replacement configuration intact"
);
});
}
}
+264 -307
View File
@@ -15,13 +15,19 @@
//! Bucket notification configuration use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{
NotificationConfiguration, NotificationConfigurationFilter, PutBucketNotificationConfigurationInput,
PutBucketNotificationConfigurationOutput,
};
use crate::app::storage_api::s3::region::Region;
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use rustfs_gateway_types::persistence::parse_notification_dto;
#[cfg(test)]
use rustfs_s3_types::S3ErrorCode;
use rustfs_s3_types::{S3Result, s3_error};
/// Runtime rules input retained alongside the prepared persisted configuration.
pub(crate) struct NotificationDestination {
pub(crate) kind: &'static str,
pub(crate) id: Option<String>,
pub(crate) events: Vec<String>,
pub(crate) target_arn: String,
pub(crate) filters: Vec<(Option<String>, Option<String>)>,
}
fn is_valid_notification_filter_value(value: &str) -> bool {
if value.len() > 1024 || value.contains('\\') {
@@ -41,30 +47,24 @@ fn invalid_filter_name_message(cfg_scope: &str, name: &str) -> String {
)
}
fn validate_notification_filter_rules(
filter: Option<&NotificationConfigurationFilter>,
cfg_kind: &str,
cfg_id: Option<&str>,
) -> S3Result<()> {
let Some(filter) = filter else {
fn validate_notification_filter_rules(destination: &NotificationDestination) -> S3Result<()> {
if destination.filters.is_empty() {
return Ok(());
};
let Some(s3key_filter) = filter.key.as_ref() else {
return Ok(());
};
let Some(rules) = s3key_filter.filter_rules.as_ref() else {
return Ok(());
};
}
let mut has_prefix = false;
let mut has_suffix = false;
let cfg_scope = cfg_id.map_or_else(|| cfg_kind.to_string(), |id| format!("{cfg_kind} id={id}"));
let kind = destination.kind;
let cfg_scope = destination
.id
.as_ref()
.map_or_else(|| kind.to_owned(), |id| format!("{kind} id={id}"));
for rule in rules {
let Some(name) = rule.name.as_ref() else {
for (name, value) in &destination.filters {
let Some(name) = name.as_deref() else {
return Err(s3_error!(InvalidArgument, "invalid notification filter rule: missing Name ({cfg_scope})"));
};
let Some(value) = rule.value.as_ref() else {
let Some(value) = value.as_deref() else {
return Err(s3_error!(
InvalidArgument,
"invalid notification filter rule: missing Value ({cfg_scope})"
@@ -75,43 +75,24 @@ fn validate_notification_filter_rules(
return Err(s3_error!(InvalidArgument, "{}", invalid_filter_value_message(&cfg_scope, value)));
}
if name.as_str().eq_ignore_ascii_case("prefix") {
if name.eq_ignore_ascii_case("prefix") {
if has_prefix {
return Err(s3_error!(InvalidArgument, "duplicate notification filter name 'prefix' ({cfg_scope})"));
}
has_prefix = true;
} else if name.as_str().eq_ignore_ascii_case("suffix") {
} else if name.eq_ignore_ascii_case("suffix") {
if has_suffix {
return Err(s3_error!(InvalidArgument, "duplicate notification filter name 'suffix' ({cfg_scope})"));
}
has_suffix = true;
} else {
return Err(s3_error!(InvalidArgument, "{}", invalid_filter_name_message(&cfg_scope, name.as_str())));
return Err(s3_error!(InvalidArgument, "{}", invalid_filter_name_message(&cfg_scope, name)));
}
}
Ok(())
}
fn validate_notification_configuration_filters(notification_configuration: &NotificationConfiguration) -> S3Result<()> {
if let Some(queue_configs) = notification_configuration.queue_configurations.as_ref() {
for cfg in queue_configs {
validate_notification_filter_rules(cfg.filter.as_ref(), "QueueConfiguration", cfg.id.as_deref())?;
}
}
if let Some(topic_configs) = notification_configuration.topic_configurations.as_ref() {
for cfg in topic_configs {
validate_notification_filter_rules(cfg.filter.as_ref(), "TopicConfiguration", cfg.id.as_deref())?;
}
}
if let Some(lambda_configs) = notification_configuration.lambda_function_configurations.as_ref() {
for cfg in lambda_configs {
validate_notification_filter_rules(cfg.filter.as_ref(), "LambdaFunctionConfiguration", cfg.id.as_deref())?;
}
}
Ok(())
}
fn parse_notification_target_id(arn_str: &str) -> Result<TargetID, TargetIDError> {
ARN::parse(arn_str)
.map(|arn| arn.target_id)
@@ -120,43 +101,34 @@ fn parse_notification_target_id(arn_str: &str) -> Result<TargetID, TargetIDError
type NotificationEventRule = (Vec<EventName>, String, String, Vec<TargetID>);
/// Builds the notify runtime rules for a bucket notification configuration
/// without touching the store or the runtime rule state.
fn build_notification_event_rules(
notification_configuration: &NotificationConfiguration,
) -> S3Result<Vec<NotificationEventRule>> {
let mut event_rules = Vec::new();
let invalid_arn = |e: TargetIDError| {
S3Error::with_message(S3ErrorCode::InvalidArgument, format!("Invalid ARN in notification configuration: {e}"))
};
process_queue_configurations(
&mut event_rules,
notification_configuration.queue_configurations.clone(),
parse_notification_target_id,
)
.map_err(invalid_arn)?;
process_topic_configurations(
&mut event_rules,
notification_configuration.topic_configurations.clone(),
parse_notification_target_id,
)
.map_err(invalid_arn)?;
process_lambda_configurations(
&mut event_rules,
notification_configuration.lambda_function_configurations.clone(),
parse_notification_target_id,
)
.map_err(invalid_arn)?;
Ok(event_rules)
fn build_notification_event_rules(destinations: Vec<NotificationDestination>) -> S3Result<Vec<NotificationEventRule>> {
destinations
.into_iter()
.map(|destination| {
let events = destination
.events
.iter()
.filter_map(|event| EventName::parse(event).ok())
.collect();
let (mut prefix, mut suffix) = (String::new(), String::new());
for (name, value) in destination.filters {
if let (Some(name), Some(value)) = (name, value) {
if name.eq_ignore_ascii_case("prefix") {
prefix = value;
} else if name.eq_ignore_ascii_case("suffix") {
suffix = value;
}
}
}
let target = parse_notification_target_id(&destination.target_arn)
.map_err(|err| s3_error!(InvalidArgument, "Invalid ARN in notification configuration: {err}"))?;
Ok((events, prefix, suffix, vec![target]))
})
.collect()
}
fn resolve_notification_region(global_region: Option<Region>, request_region: Option<Region>) -> String {
global_region
.or(request_region)
.map(|region| region.to_string())
.unwrap_or_else(|| RUSTFS_REGION.to_string())
fn resolve_notification_region(global_region: Option<String>, request_region: Option<&str>) -> String {
global_region.unwrap_or_else(|| request_region.unwrap_or(RUSTFS_REGION).to_owned())
}
impl DefaultBucketUsecase {
@@ -211,70 +183,74 @@ impl DefaultBucketUsecase {
})
}
pub async fn execute_put_bucket_notification_configuration(
/// Persists notification configuration, then updates the active rules for the bucket.
pub(crate) async fn execute_put_bucket_notification_configuration(
&self,
req: S3Request<PutBucketNotificationConfigurationInput>,
envelope: RequestEnvelope,
bucket: &str,
destinations: Vec<NotificationDestination>,
prepared: S3Result<PersistedNotificationConfiguration>,
) -> S3Result<S3Response<PutBucketNotificationConfigurationOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_region = req.region.clone();
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
) -> S3Result<gateway_dto::PutBucketNotificationConfigurationOutput> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::PutBucketNotificationConfiguration,
StatusCode::OK,
|envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let PutBucketNotificationConfigurationInput {
bucket,
notification_configuration,
..
} = req.input;
for destination in &destinations {
validate_notification_filter_rules(destination)?;
}
let persisted_notification_configuration = prepared?;
validate_notification_configuration_filters(&notification_configuration)?;
let persisted_notification_configuration = prepared?;
let Some(store) = self.object_store() else {
return Err(s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let region =
resolve_notification_region(self.global_region().map(|region| region.to_string()), envelope.region());
let notify = current_notify_interface_for_context(self.context.as_deref());
let event_rules = build_notification_event_rules(destinations)?;
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
// Validate before the write. Clear and add still run after the durable update.
notify
.validate_event_specific_rules(bucket, &region, &event_rules)
.await
.map_err(|e| s3_error!(InternalError, "Failed to add rules: {e}"))?;
let region = resolve_notification_region(self.global_region(), request_region);
let notify = current_notify_interface_for_context(self.context.as_deref());
let event_rules = build_notification_event_rules(&notification_configuration)?;
let data = serialize_config(&persisted_notification_configuration)?;
update_bucket_config_for_incarnation(bucket, BUCKET_NOTIFICATION_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
// Reject the request before the store write so a failure cannot leave a
// persisted configuration that the notify runtime refused to activate.
notify
.validate_event_specific_rules(&bucket, region.as_str(), &event_rules)
.await
.map_err(|e| s3_error!(InternalError, "Failed to add rules: {e}"))?;
notify_bucket_metadata_reload(bucket.to_owned(), "put bucket notification", request_context, false).await;
let data = serialize_config(&persisted_notification_configuration)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_NOTIFICATION_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify
.clear_bucket_notification_rules(bucket)
.await
.map_err(|e| s3_error!(InternalError, "Failed to clear rules: {e}"))?;
warn!("notify event rules: {:?}", &event_rules);
notify
.add_event_specific_rules(bucket, &region, &event_rules)
.await
.map_err(|e| s3_error!(InternalError, "Failed to add rules: {e}"))?;
notify_bucket_metadata_reload(bucket.clone(), "put bucket notification", request_context, false).await;
notify
.clear_bucket_notification_rules(&bucket)
.await
.map_err(|e| s3_error!(InternalError, "Failed to clear rules: {e}"))?;
warn!("notify event rules: {:?}", &event_rules);
notify
.add_event_specific_rules(&bucket, region.as_str(), &event_rules)
.await
.map_err(|e| s3_error!(InternalError, "Failed to add rules: {e}"))?;
Ok(S3Response::new(PutBucketNotificationConfigurationOutput {}))
Ok(gateway_dto::PutBucketNotificationConfigurationOutput::default())
},
)
.await
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::storage_api::s3::dto::{FilterRule, FilterRuleName, QueueConfiguration, S3KeyFilter};
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use http::Method;
#[tokio::test]
@@ -294,43 +270,40 @@ mod tests {
#[test]
fn resolve_notification_region_prefers_global_region() {
let binding = resolve_notification_region(Some("us-east-1".parse().unwrap()), Some("ap-southeast-1".parse().unwrap()));
assert_eq!(binding, "us-east-1");
assert_eq!(
resolve_notification_region(Some("us-east-1".to_owned()), Some("ap-southeast-1")),
"us-east-1"
);
}
#[test]
fn resolve_notification_region_falls_back_to_request_region() {
let binding = resolve_notification_region(None, Some("ap-southeast-1".parse().unwrap()));
assert_eq!(binding, "ap-southeast-1");
assert_eq!(resolve_notification_region(None, Some("ap-southeast-1")), "ap-southeast-1");
}
#[test]
fn resolve_notification_region_defaults_value() {
let binding = resolve_notification_region(None, None);
assert_eq!(binding, RUSTFS_REGION);
assert_eq!(resolve_notification_region(None, None), RUSTFS_REGION);
}
fn destination(kind: &'static str, filters: &[(Option<&str>, Option<&str>)]) -> NotificationDestination {
NotificationDestination {
kind,
id: Some("fixture".to_owned()),
events: vec!["s3:ObjectCreated:*".to_owned()],
target_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_owned(),
filters: filters
.iter()
.map(|&(name, value)| (name.map(str::to_owned), value.map(str::to_owned)))
.collect(),
}
}
#[test]
fn validate_notification_configuration_filters_rejects_invalid_filter_name() {
let raw_name = "unsupported".repeat(100);
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: Some(FilterRuleName::from(raw_name.clone())),
value: Some("uploads/".to_string()),
}]),
}),
}),
}]),
..Default::default()
};
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
let cfg = destination("QueueConfiguration", &[(Some(&raw_name), Some("uploads/"))]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
let msg = err.message().unwrap_or_default();
assert!(msg.contains("len="), "error message should include summarized length");
@@ -339,81 +312,47 @@ mod tests {
#[test]
fn validate_notification_configuration_filters_accepts_case_insensitive_filter_names() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![
FilterRule {
name: Some(FilterRuleName::from("Prefix".to_string())),
value: Some("uploads/".to_string()),
},
FilterRule {
name: Some(FilterRuleName::from("Suffix".to_string())),
value: Some(".csv".to_string()),
},
]),
}),
}),
}]),
..Default::default()
};
validate_notification_configuration_filters(&cfg).expect("capitalized filter names should be accepted");
let cfg = destination(
"QueueConfiguration",
&[(Some("Prefix"), Some("uploads/")), (Some("Suffix"), Some(".csv"))],
);
validate_notification_filter_rules(&cfg).expect("capitalized filter names should be accepted");
}
#[test]
fn validate_notification_configuration_filters_rejects_duplicate_prefix_rules() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![
FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::PREFIX)),
value: Some("uploads/".to_string()),
},
FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::PREFIX)),
value: Some("images/".to_string()),
},
]),
}),
}),
}]),
..Default::default()
};
for kind in ["QueueConfiguration"] {
for duplicate in ["prefix", "PREFIX"] {
let cfg = destination(kind, &[(Some("prefix"), Some("uploads/")), (Some(duplicate), Some("images/"))]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
err.message(),
Some(format!("duplicate notification filter name 'prefix' ({kind} id=fixture)").as_str())
);
}
}
}
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
#[test]
fn validate_notification_configuration_filters_rejects_duplicate_suffix_rules() {
for kind in ["QueueConfiguration", "TopicConfiguration"] {
for duplicate in ["suffix", "SUFFIX"] {
let cfg = destination(kind, &[(Some("suffix"), Some(".csv")), (Some(duplicate), Some(".log"))]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
err.message(),
Some(format!("duplicate notification filter name 'suffix' ({kind} id=fixture)").as_str())
);
}
}
}
#[test]
fn validate_notification_configuration_filters_rejects_invalid_filter_value() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::SUFFIX)),
value: Some("../secret".to_string()),
}]),
}),
}),
}]),
..Default::default()
};
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
let cfg = destination("QueueConfiguration", &[(Some("suffix"), Some("../secret"))]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
let msg = err.message().unwrap_or_default();
assert!(msg.contains("len="), "error message should include summarized length");
@@ -422,109 +361,127 @@ mod tests {
#[test]
fn validate_notification_configuration_filters_rejects_missing_filter_name() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: None,
value: Some("uploads/".to_string()),
}]),
}),
}),
}]),
..Default::default()
};
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
for (kind, value) in [
("QueueConfiguration", Some("uploads/")),
("LambdaFunctionConfiguration", None),
] {
let cfg = destination(kind, &[(None, value)]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
err.message(),
Some(format!("invalid notification filter rule: missing Name ({kind} id=fixture)").as_str())
);
}
}
#[test]
fn validate_notification_configuration_filters_rejects_missing_filter_value() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::PREFIX)),
value: None,
}]),
}),
}),
}]),
..Default::default()
};
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
let cfg = destination("QueueConfiguration", &[(Some("prefix"), None)]);
let err = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
err.message(),
Some("invalid notification filter rule: missing Value (QueueConfiguration id=fixture)")
);
}
#[test]
fn validate_notification_configuration_filters_rejects_duplicate_suffix_rules() {
let cfg = NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![
FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::SUFFIX)),
value: Some(".csv".to_string()),
},
FilterRule {
name: Some(FilterRuleName::from_static(FilterRuleName::SUFFIX)),
value: Some(".log".to_string()),
},
]),
}),
}),
}]),
..Default::default()
};
let err = validate_notification_configuration_filters(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
fn notification_filter_values_preserve_byte_and_path_boundaries() {
let limit = "é".repeat(512);
for value in ["", "a.b/..suffix/.prefix", &limit] {
let cfg = destination("QueueConfiguration", &[(Some("prefix"), Some(value))]);
validate_notification_filter_rules(&cfg).unwrap();
}
let too_long = format!("{limit}a");
for value in ["a\\b", "a/./b", "a/../b", &too_long] {
let cfg = destination("QueueConfiguration", &[(Some("unknown"), Some(value))]);
let error = validate_notification_filter_rules(&cfg).unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
error.message(),
Some(format!("invalid notification filter value (len={}) (QueueConfiguration id=fixture)", value.len()).as_str())
);
}
}
#[tokio::test]
async fn execute_put_bucket_notification_configuration_rejects_invalid_filter_before_store_lookup() {
let input = PutBucketNotificationConfigurationInput::builder()
.bucket("test-bucket".to_string())
.notification_configuration(NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("q1".to_string()),
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".to_string(),
events: vec!["s3:ObjectCreated:*".to_string().into()],
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![FilterRule {
name: Some(FilterRuleName::from("unsupported".to_string())),
value: Some("uploads/".to_string()),
}]),
}),
}),
}]),
..Default::default()
})
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_notification_configuration(req, Ok(PersistedNotificationConfiguration::default()))
let bucket = "test-bucket";
let error = usecase
.execute_put_bucket_notification_configuration(
owned_envelope(Method::PUT),
bucket,
vec![destination("QueueConfiguration", &[(Some("unsupported"), Some("uploads/"))])],
Ok(PersistedNotificationConfiguration::default()),
)
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(error.code(), &S3ErrorCode::InvalidArgument);
let invalid = [
destination("QueueConfiguration", &[(Some("unsupported"), Some("uploads/"))]),
destination("TopicConfiguration", &[(None, None)]),
];
let error = usecase
.execute_put_bucket_notification_configuration(
owned_envelope(Method::PUT),
bucket,
invalid.into(),
Err(s3_error!(MalformedXML, "persistence conversion")),
)
.await
.unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(
error.message(),
Some(
"invalid notification filter name (len=11) (only 'prefix'/'suffix' are supported) (QueueConfiguration id=fixture)"
)
);
let error = usecase
.execute_put_bucket_notification_configuration(
owned_envelope(Method::PUT),
bucket,
Vec::new(),
Err(s3_error!(MalformedXML, "persistence conversion")),
)
.await
.unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::MalformedXML);
assert_eq!(error.message(), Some("persistence conversion"));
let mut request = build_request((), Method::PUT);
request.extensions.insert(crate::runtime_sources::ServerContextSlot::new());
let error = usecase
.execute_put_bucket_notification_configuration(
RequestEnvelope::from_s3s(&request),
bucket,
vec![destination("QueueConfiguration", &[(None, None)])],
Err(s3_error!(MalformedXML, "persistence conversion")),
)
.await
.unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(error.message(), Some("bucket config mutation snapshot is missing"));
let mut invalid_arn = destination("QueueConfiguration", &[]);
invalid_arn.target_arn = "invalid-arn".to_owned();
let error = usecase
.execute_put_bucket_notification_configuration(
owned_envelope(Method::PUT),
bucket,
vec![invalid_arn],
Ok(PersistedNotificationConfiguration::default()),
)
.await
.unwrap_err();
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(error.message(), Some("Not init"));
}
}
#[cfg(test)]
mod side_effect_tests;
@@ -0,0 +1,459 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Store and callback observations through the legacy Put entrypoint and shared use case.
use super::*;
use crate::app::bucket_usecase::test_support::build_request_with_req_info;
use crate::app::gating_test_env::{app_context_from_current_environment, run_large_stack_test, shared_gating_ambient};
use crate::app::storage_api::bucket_usecase::access::ReqInfo;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::s3::dto::{
FilterRule, LambdaFunctionConfiguration, NotificationConfiguration, NotificationConfigurationFilter,
PutBucketNotificationConfigurationInput, QueueConfiguration, S3KeyFilter, TopicConfiguration,
};
use crate::app::storage_api::s3::{S3, S3Access, S3ErrorCode, S3Request};
use crate::app::storage_api::test::ecfs::FS;
use crate::runtime_sources::{NotifyInterface, ServerContextSlot};
use http::Method;
use rustfs_notify::{EventArgs, NotificationError};
use std::sync::Mutex;
const EMPTY_XML: &[u8] = b"<NotificationConfiguration></NotificationConfiguration>";
const OLD_XML: &[u8] =
b"<NotificationConfiguration><EventBridgeConfiguration></EventBridgeConfiguration></NotificationConfiguration>";
const NEW_XML: &[u8] = concat!(
"<NotificationConfiguration>",
"<CloudFunctionConfiguration><Event>s3:Future:Unrecognized</Event><Id>lambda</Id><CloudFunction>arn:rustfs:sqs:us-east-1:3:webhook</CloudFunction></CloudFunctionConfiguration>",
"<QueueConfiguration><Event>s3:ObjectCreated:*</Event><Event>s3:Future:Unrecognized</Event>",
"<Filter><S3Key><FilterRule><Name>Prefix</Name><Value>logs/%2F</Value></FilterRule>",
"<FilterRule><Name>Suffix</Name><Value>.csv</Value></FilterRule></S3Key></Filter>",
"<Id>queue</Id><Queue>arn:rustfs:sqs:us-east-1:1:webhook</Queue></QueueConfiguration>",
"<TopicConfiguration><Event>s3:ObjectRemoved:*</Event><Id>topic</Id><Topic>arn:rustfs:sqs:us-east-1:2:webhook</Topic></TopicConfiguration>",
"</NotificationConfiguration>"
).as_bytes();
type RuntimeRule = (Vec<EventName>, String, String, Vec<TargetID>);
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Callback {
Validate,
ReloadStarted,
ReloadCompleted,
Clear,
Add,
}
#[derive(Debug, PartialEq, Eq)]
enum Observation {
Rules(Callback, String, Vec<RuntimeRule>, Vec<u8>),
Reload(Callback, &'static str, bool),
Clear(Vec<u8>),
}
struct RecordingNotify {
bucket: String,
fail_at: Option<Callback>,
calls: Arc<Mutex<Vec<Observation>>>,
active_rules: Mutex<Vec<RuntimeRule>>,
}
async fn read_notification_xml(bucket: &str) -> Vec<u8> {
crate::app::storage_api::test::get_global_bucket_metadata_sys()
.expect("test bucket metadata")
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("load notification bytes from disk")
.notification_config_xml
}
impl RecordingNotify {
fn maybe_fail(&self, callback: Callback) -> Result<(), NotificationError> {
if self.fail_at == Some(callback) {
return Err(NotificationError::RuleConfiguration("fixture failure".into()));
}
Ok(())
}
async fn record_rules(&self, callback: Callback, bucket: &str, region: &str, rules: &[RuntimeRule]) {
assert_eq!(bucket, self.bucket);
let xml = read_notification_xml(bucket).await;
self.calls
.lock()
.unwrap()
.push(Observation::Rules(callback, region.to_owned(), rules.to_vec(), xml));
}
}
#[async_trait::async_trait]
impl NotifyInterface for RecordingNotify {
async fn notify(&self, _args: EventArgs) {
panic!("bucket notification configuration must not emit an object event");
}
async fn validate_event_specific_rules(
&self,
bucket: &str,
region: &str,
rules: &[RuntimeRule],
) -> Result<(), NotificationError> {
self.record_rules(Callback::Validate, bucket, region, rules).await;
self.maybe_fail(Callback::Validate)
}
async fn add_event_specific_rules(&self, bucket: &str, region: &str, rules: &[RuntimeRule]) -> Result<(), NotificationError> {
self.record_rules(Callback::Add, bucket, region, rules).await;
self.maybe_fail(Callback::Add)?;
*self.active_rules.lock().unwrap() = rules.to_vec();
Ok(())
}
async fn clear_bucket_notification_rules(&self, bucket: &str) -> Result<(), NotificationError> {
assert_eq!(bucket, self.bucket);
let xml = read_notification_xml(bucket).await;
self.calls.lock().unwrap().push(Observation::Clear(xml));
self.maybe_fail(Callback::Clear)?;
self.active_rules.lock().unwrap().clear();
Ok(())
}
}
fn configuration() -> NotificationConfiguration {
NotificationConfiguration {
queue_configurations: Some(vec![QueueConfiguration {
id: Some("queue".into()),
events: vec![
"s3:ObjectCreated:*".to_owned().into(),
"s3:Future:Unrecognized".to_owned().into(),
],
queue_arn: "arn:rustfs:sqs:us-east-1:1:webhook".into(),
filter: Some(NotificationConfigurationFilter {
key: Some(S3KeyFilter {
filter_rules: Some(vec![
FilterRule {
name: Some("Prefix".to_owned().into()),
value: Some("logs/%2F".into()),
},
FilterRule {
name: Some("Suffix".to_owned().into()),
value: Some(".csv".into()),
},
]),
}),
}),
}]),
topic_configurations: Some(vec![TopicConfiguration {
id: Some("topic".into()),
events: vec!["s3:ObjectRemoved:*".to_owned().into()],
topic_arn: "arn:rustfs:sqs:us-east-1:2:webhook".into(),
filter: None,
}]),
lambda_function_configurations: Some(vec![LambdaFunctionConfiguration {
id: Some("lambda".into()),
events: vec!["s3:Future:Unrecognized".to_owned().into()],
lambda_function_arn: "arn:rustfs:sqs:us-east-1:3:webhook".into(),
filter: None,
}]),
..Default::default()
}
}
fn expected_rules() -> Vec<RuntimeRule> {
[
(vec![EventName::parse("s3:ObjectCreated:*").unwrap()], "logs/%2F", ".csv", "1"),
(vec![EventName::parse("s3:ObjectRemoved:*").unwrap()], "", "", "2"),
(Vec::new(), "", "", "3"),
]
.map(|(events, prefix, suffix, id)| {
(
events,
prefix.to_owned(),
suffix.to_owned(),
vec![TargetID {
id: id.to_owned(),
name: "webhook".into(),
}],
)
})
.into()
}
fn notification_request(
bucket: &str,
configuration: NotificationConfiguration,
slot: &Arc<ServerContextSlot>,
) -> S3Request<PutBucketNotificationConfigurationInput> {
let input = PutBucketNotificationConfigurationInput::builder()
.bucket(bucket.to_owned())
.notification_configuration(configuration)
.build()
.unwrap();
let mut request = build_request_with_req_info(
input,
Method::PUT,
ReqInfo {
bucket: Some(bucket.to_owned()),
..Default::default()
},
);
request.extensions.insert(slot.clone());
request.uri = format!("/{bucket}?notification").parse().unwrap();
request.region = Some("eu-west-2".parse().unwrap());
request
}
fn check_notification_write_order() {
run_large_stack_test("notification-write-order", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
for fail_at in [None, Some(Callback::Validate), Some(Callback::Clear), Some(Callback::Add)] {
let bucket = format!("notify-order-{}", uuid::Uuid::new_v4().simple());
store.make_bucket(&bucket, &MakeBucketOptions::default()).await.unwrap();
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, OLD_XML.to_vec())
.await
.unwrap();
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Principal": {"AWS": "*"}, "Action": ["s3:PutBucketNotification"],
"Resource": [format!("arn:aws:s3:::{bucket}")]}]
});
store
.update_bucket_metadata_config(&bucket, BUCKET_POLICY_CONFIG, serde_json::to_vec(&policy).unwrap())
.await
.unwrap();
let calls = Arc::new(Mutex::new(Vec::new()));
let recorder = Arc::new(RecordingNotify {
bucket: bucket.clone(),
fail_at,
calls: calls.clone(),
active_rules: Mutex::new(expected_rules()),
});
let context = Arc::new(app_context_from_current_environment(&ambient).with_test_notify_interface(recorder.clone()));
let region = context
.region()
.get()
.map_or_else(|| "eu-west-2".to_owned(), |region| region.to_string());
let slot = ServerContextSlot::new();
assert!(slot.install(context.clone()));
let fs = FS::with_server_ctx(slot.clone());
let audit = MutationCompletionProbe::install(&bucket);
let mut request = notification_request(&bucket, configuration(), &slot);
S3Access::put_bucket_notification_configuration(&fs, &mut request)
.await
.unwrap();
let result = observe_reload(&bucket, calls.clone(), S3::put_bucket_notification_configuration(&fs, request)).await;
let expected = expected_rules();
let mut timeline = vec![Observation::Rules(
Callback::Validate,
region.clone(),
expected.clone(),
OLD_XML.to_vec(),
)];
if fail_at != Some(Callback::Validate) {
timeline.extend([
Observation::Reload(Callback::ReloadStarted, "put bucket notification", false),
Observation::Reload(Callback::ReloadCompleted, "put bucket notification", false),
Observation::Clear(NEW_XML.to_vec()),
]);
if fail_at != Some(Callback::Clear) {
timeline.push(Observation::Rules(Callback::Add, region.clone(), expected.clone(), NEW_XML.to_vec()));
}
}
assert_eq!(*calls.lock().unwrap(), timeline);
let stored = if fail_at == Some(Callback::Validate) {
OLD_XML
} else {
NEW_XML
};
assert_eq!(read_notification_xml(&bucket).await, stored);
let error_message = match fail_at {
Some(Callback::Clear) => Some("Failed to clear rules: Rule configuration error: fixture failure"),
Some(_) => Some("Failed to add rules: Rule configuration error: fixture failure"),
None => None,
};
if let Some(message) = error_message {
let error = result.expect_err("injected callback failure");
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(error.message(), Some(message));
} else {
result.expect("configuration written");
}
{
let audits = audit.audits.lock().unwrap();
let [entry] = audits.as_slice() else { panic!("one notification audit: {audits:?}") };
assert_eq!(entry.api.name.as_deref(), Some(S3Operation::PutBucketNotificationConfiguration.as_str()));
assert_eq!(entry.api.bucket.as_deref(), Some(bucket.as_str()));
assert_eq!(entry.api.status_code, Some(if fail_at.is_some() { 500 } else { 200 }));
assert_eq!(entry.error.as_deref().filter(|message| !message.is_empty()), error_message);
}
assert!(audit.events.lock().unwrap().is_empty());
if fail_at.is_none() {
assert_eq!(*recorder.active_rules.lock().unwrap(), expected);
let mut invalid = configuration();
invalid.lambda_function_configurations.as_mut().unwrap()[0].lambda_function_arn = "invalid-arn".into();
let mut invalid = notification_request(&bucket, invalid, &slot);
S3Access::put_bucket_notification_configuration(&fs, &mut invalid)
.await
.unwrap();
calls.lock().unwrap().clear();
let err = observe_reload(&bucket, calls.clone(), S3::put_bucket_notification_configuration(&fs, invalid))
.await
.expect_err("the last destination has an invalid ARN");
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert!(
err.message()
.unwrap()
.starts_with("Invalid ARN in notification configuration: ")
);
assert!(
calls.lock().unwrap().is_empty(),
"validation happens after all destinations have been built"
);
assert_eq!(read_notification_xml(&bucket).await, NEW_XML);
let usecase = DefaultBucketUsecase::with_context(Some(context));
let output = usecase
.execute_get_bucket_notification_configuration(gateway_dto::GetBucketNotificationConfigurationInput {
bucket: super::super::test_support::gateway_bucket(&bucket),
..Default::default()
})
.await
.unwrap();
assert_eq!(output.queue_configurations[0].events[1].as_str(), "s3:Future:Unrecognized");
calls.lock().unwrap().clear();
let mut empty = notification_request(&bucket, NotificationConfiguration::default(), &slot);
S3Access::put_bucket_notification_configuration(&fs, &mut empty)
.await
.unwrap();
observe_reload(&bucket, calls.clone(), S3::put_bucket_notification_configuration(&fs, empty))
.await
.unwrap();
assert_eq!(
*calls.lock().unwrap(),
[
Observation::Rules(Callback::Validate, region.clone(), Vec::new(), NEW_XML.to_vec()),
Observation::Reload(Callback::ReloadStarted, "put bucket notification", false),
Observation::Reload(Callback::ReloadCompleted, "put bucket notification", false),
Observation::Clear(EMPTY_XML.to_vec()),
Observation::Rules(Callback::Add, region, Vec::new(), EMPTY_XML.to_vec()),
]
);
assert_eq!(read_notification_xml(&bucket).await, EMPTY_XML);
assert!(recorder.active_rules.lock().unwrap().is_empty());
}
}
});
}
#[test]
#[serial_test::serial]
fn legacy_put_notification_keeps_callback_order_and_durable_bytes() {
temp_env::with_var(rustfs_config::ENV_AUDIT_ENABLE, Some("true"), || {
crate::server::refresh_audit_module_enabled();
check_notification_write_order();
});
crate::server::refresh_audit_module_enabled();
}
async fn observe_reload<T>(bucket: &str, calls: Arc<Mutex<Vec<Observation>>>, operation: impl Future<Output = T>) -> T {
super::super::BUCKET_METADATA_RELOAD_PROBE
.scope(
(
bucket.to_owned(),
Box::new(move |completed, operation, scanner| {
let callback = if completed {
Callback::ReloadCompleted
} else {
Callback::ReloadStarted
};
calls.lock().unwrap().push(Observation::Reload(callback, operation, scanner));
}),
),
operation,
)
.await
}
#[test]
fn legacy_put_notification_does_not_write_after_the_bucket_is_recreated() {
run_large_stack_test("notification-stale-incarnation", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let bucket = format!("notify-stale-{}", uuid::Uuid::new_v4().simple());
store.make_bucket(&bucket, &MakeBucketOptions::default()).await.unwrap();
let first = store.bucket_incarnation_id(&bucket).await.unwrap();
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{"Effect": "Allow", "Principal": {"AWS": "*"}, "Action": ["s3:PutBucketNotification"],
"Resource": [format!("arn:aws:s3:::{bucket}")]}]
});
store
.update_bucket_metadata_config(&bucket, BUCKET_POLICY_CONFIG, serde_json::to_vec(&policy).unwrap())
.await
.unwrap();
let calls = Arc::new(Mutex::new(Vec::new()));
let recorder = Arc::new(RecordingNotify {
bucket: bucket.clone(),
fail_at: None,
calls: calls.clone(),
active_rules: Mutex::new(Vec::new()),
});
let context = Arc::new(app_context_from_current_environment(&ambient).with_test_notify_interface(recorder));
let slot = ServerContextSlot::new();
assert!(slot.install(context));
let fs = FS::with_server_ctx(slot.clone());
let mut request = notification_request(&bucket, configuration(), &slot);
S3Access::put_bucket_notification_configuration(&fs, &mut request)
.await
.unwrap();
assert_eq!(bucket_config_mutation_incarnation(&request, &bucket).unwrap(), Some(first));
let mismatch = DefaultBucketUsecase::without_context()
.execute_put_bucket_notification_configuration(
RequestEnvelope::from_s3s(&request),
"other-bucket",
vec![NotificationDestination {
kind: "QueueConfiguration",
id: None,
events: Vec::new(),
target_arn: String::new(),
filters: vec![(None, None)],
}],
Err(rustfs_s3_types::s3_error!(MalformedXML, "persistence conversion")),
)
.await
.unwrap_err();
assert_eq!(mismatch.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
assert_eq!(mismatch.message(), Some("bucket config mutation snapshot does not match request bucket"));
store.delete_bucket(&bucket, &DeleteBucketOptions::default()).await.unwrap();
store.make_bucket(&bucket, &MakeBucketOptions::default()).await.unwrap();
assert_ne!(first, store.bucket_incarnation_id(&bucket).await.unwrap());
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, OLD_XML.to_vec())
.await
.unwrap();
let result = observe_reload(&bucket, calls.clone(), S3::put_bucket_notification_configuration(&fs, request)).await;
assert_eq!(read_notification_xml(&bucket).await, OLD_XML);
assert_eq!(result.unwrap_err().code(), &S3ErrorCode::NoSuchBucket);
assert!(
matches!(calls.lock().unwrap().as_slice(), [Observation::Rules(Callback::Validate, _, _, xml)] if xml == OLD_XML)
);
});
}
+560 -2
View File
@@ -14,14 +14,213 @@
//! Bucket Object Lock configuration use-case operations.
//!
//! NOT responsible for: the lock validators, which object writes share (`storage::ecfs_extend`).
//! Default-retention policy here applies to bucket configuration. Object writes enforce their
//! own retention rules.
use super::*;
use rustfs_gateway_types::persistence::parse_object_lock_dto;
use crate::app::storage_api::bucket_usecase::bucket::VersioningConfigExt as _;
use crate::app::storage_api::bucket_usecase::bucket::metadata::OBJECT_LOCK_CONFIG;
use rustfs_gateway_types::persistence::{PersistedDefaultRetention, parse_object_lock_dto};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
const MAXIMUM_RETENTION_DAYS: i32 = 36_500;
const MAXIMUM_RETENTION_YEARS: i32 = 100;
fn invalid_object_lock_configuration(message: impl Into<String>) -> S3Error {
S3Error::with_message(S3ErrorCode::MalformedXML, message.into())
}
fn propagate_object_lock_peer_reload(result: std::result::Result<(), StorageError>) -> S3Result<()> {
result.map_err(|err| {
S3Error::with_message(S3ErrorCode::InternalError, format!("Failed to publish Object Lock metadata: {err}"))
})
}
fn invalid_retention_period(message: impl Into<String>) -> S3Error {
let mut err = S3Error::with_message(S3ErrorCode::Custom("InvalidRetentionPeriod".into()), message.into());
err.set_status_code(StatusCode::BAD_REQUEST);
err
}
fn validate_default_retention_configuration(default_retention: &PersistedDefaultRetention) -> S3Result<()> {
let Some(mode) = default_retention.mode.as_deref() else {
return Err(invalid_object_lock_configuration("retention mode must be specified"));
};
match mode {
"COMPLIANCE" | "GOVERNANCE" => {}
_ => {
return Err(invalid_object_lock_configuration(format!("unknown retention mode {mode}")));
}
}
match (default_retention.days, default_retention.years) {
(Some(days), None) => {
if days <= 0 {
return Err(invalid_retention_period(
"Default retention period must be a positive integer value for 'Days'",
));
}
if days > MAXIMUM_RETENTION_DAYS {
return Err(invalid_retention_period(format!("Default retention period too large for 'Days' {days}",)));
}
}
(None, Some(years)) => {
if years <= 0 {
return Err(invalid_retention_period(
"Default retention period must be a positive integer value for 'Years'",
));
}
if years > MAXIMUM_RETENTION_YEARS {
return Err(invalid_retention_period(format!(
"Default retention period too large for 'Years' {years}",
)));
}
}
(Some(_), Some(_)) => {
return Err(invalid_object_lock_configuration("either Days or Years must be specified, not both"));
}
(None, None) => {
return Err(invalid_object_lock_configuration("either Days or Years must be specified"));
}
}
Ok(())
}
fn validate_object_lock_configuration_input(input_cfg: &ObjectLockConfiguration) -> S3Result<()> {
let enabled = input_cfg.object_lock_enabled.as_deref();
if enabled != Some("Enabled") {
return Err(invalid_object_lock_configuration(
"only 'Enabled' value is allowed to ObjectLockEnabled element",
));
}
if let Some(rule) = input_cfg.rule.as_ref() {
let Some(default_retention) = rule.default_retention.as_ref() else {
return Err(invalid_object_lock_configuration("Rule must include DefaultRetention"));
};
validate_default_retention_configuration(default_retention)?;
}
Ok(())
}
impl DefaultBucketUsecase {
/// Persists the lock configuration, then enables versioning under the same authorization.
///
/// Each write is fenced independently. A versioning write or peer reload can fail after
/// the lock configuration is durable.
pub(crate) async fn execute_put_object_lock_configuration(
&self,
envelope: RequestEnvelope,
bucket: &str,
config: Option<ObjectLockConfiguration>,
) -> S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutObjectLockConfiguration, StatusCode::OK, |envelope| {
self.put_object_lock_configuration(envelope, bucket, config)
})
.await
}
async fn put_object_lock_configuration(
&self,
envelope: RequestEnvelope,
bucket: &str,
config: Option<ObjectLockConfiguration>,
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let Some(config) = config else {
return Err(s3_error!(InvalidArgument, "Invalid argument."));
};
self.ensure_config_bucket(bucket).await?;
validate_object_lock_configuration_input(&config)?;
match metadata_sys::get_object_lock_config(bucket).await {
Ok(_) => {}
Err(StorageError::ConfigNotFound) => {
if !BucketVersioningSys::enabled(bucket).await {
return Err(s3_error!(
InvalidBucketState,
"Object Lock configuration cannot be enabled on existing buckets"
));
}
}
Err(err) => {
warn!(
component = LOG_COMPONENT_APP,
subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
event = "object_lock_config_load_failed",
bucket = %bucket,
error = ?err,
"Failed to load bucket object lock configuration"
);
return Err(s3_error!(InternalError, "Failed to get bucket ObjectLockConfiguration"));
}
};
let data = serialize_config(&config)?;
let object_lock_config = String::from_utf8(data.clone()).map_err(to_internal_error)?;
let updated_at = update_bucket_config_for_incarnation(bucket, OBJECT_LOCK_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
#[cfg(test)]
tests::after_lock_write(bucket).await;
// Versioning is a separate write and must stay fenced to the original incarnation.
let versioning_config = BucketVersioningSys::get(bucket).await.map_err(ApiError::from)?;
if !versioning_config.enabled() {
let versioning_config = PersistedVersioningConfiguration {
status: Some("Enabled".to_owned()),
..Default::default()
};
let data = serialize_config(&versioning_config)?;
update_bucket_config_for_incarnation(bucket, BUCKET_VERSIONING_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
}
let peer_reload = match current_notification_system_for_context(self.context.as_deref()) {
Some(notification_sys) => notification_sys.load_bucket_metadata(bucket).await,
None => Ok(()),
};
#[cfg(test)]
let peer_reload = peer_reload.and(tests::peer_reload_outcome(bucket).await);
propagate_object_lock_peer_reload(peer_reload)?;
let site_result = site_replication_bucket_meta_hook(SRBucketMeta {
bucket: bucket.to_owned(),
r#type: "object-lock-config".to_owned(),
object_lock_config: Some(object_lock_config),
updated_at: Some(updated_at),
api_version: Some(SITE_REPL_API_VERSION.to_owned()),
..Default::default()
})
.await;
#[cfg(test)]
let site_result = site_result.and_then(|()| tests::site_hook_outcome());
if let Err(err) = site_result {
warn!(
component = LOG_COMPONENT_APP,
subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
event = "put_object_lock_configuration",
bucket = %bucket,
result = "site_replication_hook_failed",
error = ?err,
"storage object lock state"
);
}
rustfs_scanner::record_dirty_usage_bucket_from_producer(
bucket,
rustfs_scanner::SegmentInvalidationProducerIdentity::BucketMetadata,
);
Ok(())
}
/// The stored Object Lock configuration as the gateway's answer holds it.
///
/// # Errors
@@ -75,6 +274,365 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use rustfs_gateway_types::persistence::PersistedObjectLockRule;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::owned_parity::authorize_bucket_config_mutation;
use crate::site_replication::hooks::BUCKET_META_HOOK_CALLS;
use http::Method;
use rustfs_credentials::Credentials;
use serial_test::serial;
use std::cell::{Cell, RefCell};
const ORIGINAL: &[u8] = b"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled></ObjectLockConfiguration>";
const CHANGED: &[u8] = b"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled><Rule><DefaultRetention><Days>3</Days><Mode>GOVERNANCE</Mode></DefaultRetention></Rule></ObjectLockConfiguration>";
const SUSPENDED: &[u8] = b"<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>";
const ENABLED: &[u8] = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>";
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum Scenario {
FirstFence,
SecondFence,
PeerFailure,
SiteFailure,
Success,
}
type MetadataBytes = (Vec<u8>, Vec<u8>);
struct CompletionProbe {
scenario: Scenario,
store: Arc<ECStore>,
observations: RefCell<Vec<MetadataBytes>>,
recreated_scanner_generation: Cell<Option<u64>>,
}
tokio::task_local! {
static STATE: CompletionProbe;
}
async fn stored_lock_and_versioning(bucket: &str) -> MetadataBytes {
let metadata = metadata_sys::get_global_bucket_metadata_sys()
.expect("metadata system")
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("read metadata from disk");
(metadata.object_lock_config_xml, metadata.versioning_config_xml)
}
async fn recreate(store: &Arc<ECStore>, bucket: &str) {
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
seed(bucket).await;
}
async fn seed(bucket: &str) {
metadata_sys::update(bucket, OBJECT_LOCK_CONFIG, ORIGINAL.to_vec())
.await
.expect("store original lock");
metadata_sys::update(bucket, BUCKET_VERSIONING_CONFIG, SUSPENDED.to_vec())
.await
.expect("store suspended versioning");
}
pub(super) async fn after_lock_write(bucket: &str) {
if let Ok((Scenario::SecondFence, store)) = STATE.try_with(|state| (state.scenario, Arc::clone(&state.store))) {
let bytes = stored_lock_and_versioning(bucket).await;
STATE.with(|state| state.observations.borrow_mut().push(bytes));
recreate(&store, bucket).await;
STATE.with(|state| {
state
.recreated_scanner_generation
.set(rustfs_scanner::scanner_dirty_usage_bucket_generation(bucket));
});
}
}
pub(super) async fn peer_reload_outcome(bucket: &str) -> Result<(), StorageError> {
let Ok(scenario) = STATE.try_with(|state| state.scenario) else {
return Ok(());
};
let bytes = stored_lock_and_versioning(bucket).await;
STATE.with(|state| state.observations.borrow_mut().push(bytes));
if scenario == Scenario::PeerFailure {
return Err(StorageError::other("test peer reload failure"));
}
Ok(())
}
pub(super) fn site_hook_outcome() -> S3Result<()> {
if matches!(STATE.try_with(|state| state.scenario), Ok(Scenario::SiteFailure)) {
return Err(s3_error!(InternalError, "test site replication failure"));
}
Ok(())
}
async fn put_lock_with_observed_completion(scenario: Scenario) {
let context = shared_gating_ambient().await;
let store = context.object_store();
let bucket = format!(
"lock-cfg-{}-{}",
match scenario {
Scenario::FirstFence => "first",
Scenario::SecondFence => "second",
Scenario::PeerFailure => "peer",
Scenario::SiteFailure => "site",
Scenario::Success => "success",
},
uuid::Uuid::new_v4().simple()
);
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
seed(&bucket).await;
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(&context)));
let mut request = build_request((), Method::PUT);
request.uri = format!("/{bucket}?object-lock").parse().expect("request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "bucket-lock-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(
&server,
&mut envelope,
&bucket,
Action::S3Action(S3Action::PutBucketObjectLockConfigurationAction),
)
.await
.expect("capture the authorized bucket incarnation");
if scenario == Scenario::FirstFence {
recreate(&store, &bucket).await;
}
let scanner_before = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let audits = MutationCompletionProbe::install(&bucket);
let usecase = DefaultBucketUsecase::with_context(Some(context));
let config = rustfs_gateway_types::persistence::parse_object_lock(CHANGED).expect("fixture parses");
let (result, hooks, observed, recreated_scanner_generation) = STATE
.scope(
CompletionProbe {
scenario,
store,
observations: RefCell::default(),
recreated_scanner_generation: Cell::default(),
},
async {
let (result, hooks) = BUCKET_META_HOOK_CALLS
.scope(RefCell::default(), async {
let result = usecase
.execute_put_object_lock_configuration(envelope, &bucket, Some(config))
.await;
(result, BUCKET_META_HOOK_CALLS.with(RefCell::take))
})
.await;
let (observed, recreated_scanner_generation) =
STATE.with(|state| (state.observations.take(), state.recreated_scanner_generation.get()));
(result, hooks, observed, recreated_scanner_generation)
},
)
.await;
let stored = stored_lock_and_versioning(&bucket).await;
let scanner_after = rustfs_scanner::scanner_dirty_usage_bucket_generation(&bucket);
let audits_seen = audits.audits.lock().expect("audit entries");
let seen: Vec<_> = audits_seen
.iter()
.map(|entry| (entry.api.name.as_deref(), entry.api.status_code))
.collect();
let batch = |lock: &[u8], version: &[u8]| (lock.to_vec(), version.to_vec());
let (error, versioning, callbacks, status) = match scenario {
Scenario::FirstFence => (Some(S3ErrorCode::NoSuchBucket), batch(ORIGINAL, SUSPENDED), vec![], 404),
Scenario::SecondFence => (
Some(S3ErrorCode::NoSuchBucket),
batch(ORIGINAL, SUSPENDED),
vec![batch(CHANGED, SUSPENDED)],
404,
),
Scenario::PeerFailure => (
Some(S3ErrorCode::InternalError),
batch(CHANGED, ENABLED),
vec![batch(CHANGED, ENABLED)],
500,
),
Scenario::SiteFailure | Scenario::Success => (None, batch(CHANGED, ENABLED), vec![batch(CHANGED, ENABLED)], 200),
};
assert_eq!(
(result.err().map(|error| error.code().clone()), stored, observed, seen),
(
error,
versioning,
callbacks,
vec![(Some(S3Operation::PutObjectLockConfiguration.as_str()), Some(status))]
),
"{scenario:?}",
);
if scenario == Scenario::SiteFailure || scenario == Scenario::Success {
let [hook] = hooks.as_slice() else { panic!("{scenario:?} must call site replication once") };
assert_eq!(
(&hook.bucket, hook.item_type.as_str(), hook.scanner_generation),
(&bucket, "object-lock-config", scanner_before)
);
assert!(scanner_after > scanner_before, "{scenario:?} must invalidate scanner usage");
} else {
assert!(hooks.is_empty(), "{scenario:?} must stop before site replication");
let expected_generation = if scenario == Scenario::SecondFence {
recreated_scanner_generation
} else {
scanner_before
};
assert_eq!(expected_generation, scanner_after, "{scenario:?} must stop before scanner invalidation");
}
assert!(audits.events.lock().expect("event entries").is_empty());
}
#[test]
#[serial]
fn put_lock_configuration_fences_the_first_write() {
run_large_stack_test("bucket-lock-first-write", || put_lock_with_observed_completion(Scenario::FirstFence));
}
#[test]
#[serial]
fn put_lock_configuration_fences_the_versioning_write() {
run_large_stack_test("bucket-lock-version-write", || put_lock_with_observed_completion(Scenario::SecondFence));
}
#[test]
#[serial]
fn put_lock_configuration_propagates_peer_reload_failure() {
run_large_stack_test("bucket-lock-peer-failure", || put_lock_with_observed_completion(Scenario::PeerFailure));
}
#[test]
#[serial]
fn put_lock_configuration_continues_after_site_hook_failure() {
run_large_stack_test("bucket-lock-site-failure", || put_lock_with_observed_completion(Scenario::SiteFailure));
}
#[test]
#[serial]
fn put_lock_configuration_completes_durable_writes_before_callbacks() {
run_large_stack_test("bucket-lock-callback-order", || put_lock_with_observed_completion(Scenario::Success));
}
#[test]
fn test_validate_object_lock_configuration_rejects_disabled_status() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Disabled".to_owned()),
rule: None,
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::MalformedXML);
}
#[test]
fn test_validate_object_lock_configuration_rejects_invalid_default_retention_mode() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(PersistedObjectLockRule {
default_retention: Some(PersistedDefaultRetention {
mode: Some("abc".to_owned()),
days: Some(1),
years: None,
}),
}),
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::MalformedXML);
}
#[test]
fn test_validate_object_lock_configuration_rejects_days_and_years_together() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(PersistedObjectLockRule {
default_retention: Some(PersistedDefaultRetention {
mode: Some("GOVERNANCE".to_owned()),
days: Some(1),
years: Some(1),
}),
}),
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::MalformedXML);
}
#[test]
fn test_validate_object_lock_configuration_rejects_missing_default_retention() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(PersistedObjectLockRule { default_retention: None }),
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::MalformedXML);
}
#[test]
fn test_validate_object_lock_configuration_rejects_zero_days() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(PersistedObjectLockRule {
default_retention: Some(PersistedDefaultRetention {
mode: Some("GOVERNANCE".to_owned()),
days: Some(0),
years: None,
}),
}),
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::Custom("InvalidRetentionPeriod".into()));
}
#[test]
fn test_validate_object_lock_configuration_rejects_too_many_years() {
let cfg = ObjectLockConfiguration {
object_lock_enabled: Some("Enabled".to_owned()),
rule: Some(PersistedObjectLockRule {
default_retention: Some(PersistedDefaultRetention {
mode: Some("COMPLIANCE".to_owned()),
days: None,
years: Some(101),
}),
}),
};
let err = validate_object_lock_configuration_input(&cfg).unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::Custom("InvalidRetentionPeriod".into()));
}
#[test]
fn test_object_lock_peer_reload_failure_is_returned_to_the_client() {
let err = propagate_object_lock_peer_reload(Err(StorageError::other("peer reload failed")))
.expect_err("peer reload failure must prevent a successful Object Lock update response");
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert!(err.message().is_some_and(|message| message.contains("peer reload failed")));
}
#[tokio::test]
async fn execute_get_object_lock_configuration_returns_internal_error_when_store_uninitialized() {
let input = gateway_dto::GetObjectLockConfigurationInput {
+92 -84
View File
@@ -15,43 +15,42 @@
//! Bucket policy and policy status use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{
DeleteBucketPolicyInput, DeleteBucketPolicyOutput, PutBucketPolicyInput, PutBucketPolicyOutput,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use rustfs_policy::policy::Validator;
use rustfs_s3_types::s3_error;
impl DefaultBucketUsecase {
pub async fn execute_delete_bucket_policy(
pub(crate) async fn execute_delete_bucket_policy(
&self,
req: S3Request<DeleteBucketPolicyInput>,
) -> S3Result<S3Response<DeleteBucketPolicyOutput>> {
record_s3_op(S3Operation::DeleteBucketPolicy);
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeleteBucketPolicyInput { bucket, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::DeleteBucketPolicy, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(&bucket, BUCKET_POLICY_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(bucket, BUCKET_POLICY_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "delete bucket policy", request_context, false).await;
notify_bucket_metadata_reload(bucket.to_owned(), "delete bucket policy", request_context, false).await;
let item = sr_bucket_meta_item(bucket.clone(), "policy");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket policy delete hook failed");
}
let item = sr_bucket_meta_item(bucket.to_owned(), "policy");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket policy delete hook failed");
}
Ok(S3Response::new(DeleteBucketPolicyOutput {}))
Ok(())
})
.await
}
/// The stored policy document, verbatim.
@@ -166,75 +165,78 @@ impl DefaultBucketUsecase {
})
}
pub async fn execute_put_bucket_policy(
pub(crate) async fn execute_put_bucket_policy(
&self,
req: S3Request<PutBucketPolicyInput>,
) -> S3Result<S3Response<PutBucketPolicyOutput>> {
record_s3_op(S3Operation::PutBucketPolicy);
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let PutBucketPolicyInput { bucket, policy, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
policy: &str,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutBucketPolicy, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let cfg: BucketPolicy =
serde_json::from_str(&policy).map_err(|e| s3_error!(InvalidArgument, "parse policy failed {:?}", e))?;
let cfg: BucketPolicy =
serde_json::from_str(policy).map_err(|e| s3_error!(InvalidArgument, "parse policy failed {:?}", e))?;
if let Err(err) = cfg.is_valid() {
warn!("put_bucket_policy err input {:?}, {:?}", &policy, err);
return Err(s3_error!(MalformedPolicy));
}
if cfg.is_valid().is_err() {
return Err(s3_error!(MalformedPolicy));
}
let is_public_policy = cfg
.statements
.iter()
.any(|statement| matches!(statement.effect, Effect::Allow) && statement.principal.is_match("*"));
let is_public_policy = cfg
.statements
.iter()
.any(|statement| matches!(statement.effect, Effect::Allow) && statement.principal.is_match("*"));
if is_public_policy {
match metadata_sys::get_public_access_block_config(&bucket).await {
Ok((config, _)) => {
if config.block_public_policy.unwrap_or(false) {
return Err(s3_error!(AccessDenied, "Access Denied"));
if is_public_policy {
match metadata_sys::get_public_access_block_config(bucket).await {
Ok((config, _)) => {
if config.block_public_policy.unwrap_or(false) {
return Err(s3_error!(AccessDenied, "Access Denied"));
}
}
}
Err(err) => {
if err != StorageError::ConfigNotFound {
warn!(
component = LOG_COMPONENT_APP,
subsystem = LOG_SUBSYSTEM_BUCKET,
event = "bucket_public_access_block_load_failed",
bucket = %bucket,
error = ?err,
"Failed to load bucket public access block configuration"
);
return Err(ApiError::from(err).into());
Err(err) => {
if err != StorageError::ConfigNotFound {
warn!(
component = LOG_COMPONENT_APP,
subsystem = LOG_SUBSYSTEM_BUCKET,
event = "bucket_public_access_block_load_failed",
bucket = %bucket,
error = ?err,
"Failed to load bucket public access block configuration"
);
return Err(ApiError::from(err).into());
}
}
}
}
}
let data = policy.as_bytes().to_vec();
let data = policy.as_bytes().to_vec();
update_bucket_config_for_incarnation(&bucket, BUCKET_POLICY_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
update_bucket_config_for_incarnation(bucket, BUCKET_POLICY_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "put bucket policy", request_context, false).await;
notify_bucket_metadata_reload(bucket.to_owned(), "put bucket policy", request_context, false).await;
let mut item = sr_bucket_meta_item(bucket.clone(), "policy");
item.policy = Some(serde_json::from_str(&policy).map_err(|e| s3_error!(InvalidArgument, "parse policy failed {:?}", e))?);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket policy hook failed");
}
let mut item = sr_bucket_meta_item(bucket.to_owned(), "policy");
item.policy =
Some(serde_json::from_str(policy).map_err(|e| s3_error!(InvalidArgument, "parse policy failed {:?}", e))?);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket policy hook failed");
}
Ok(S3Response::new(PutBucketPolicyOutput {}))
Ok(())
})
.await
}
}
@@ -242,6 +244,8 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::{build_request, s3_op_total};
use crate::app::storage_api::s3::S3ErrorCode;
use crate::app::storage_api::s3::dto::{DeleteBucketPolicyInput, PutBucketPolicyInput};
use http::Method;
#[tokio::test]
@@ -252,10 +256,12 @@ mod tests {
.unwrap();
let req = build_request(input, Method::DELETE);
let usecase = DefaultBucketUsecase::without_context();
let fs = crate::storage::ecfs::FS::new();
let before = s3_op_total(S3Operation::DeleteBucketPolicy);
let err = usecase.execute_delete_bucket_policy(req).await.unwrap_err();
let err = crate::app::storage_api::s3::S3::delete_bucket_policy(&fs, req)
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert_eq!(s3_op_total(S3Operation::DeleteBucketPolicy), before + 1);
}
@@ -301,10 +307,12 @@ mod tests {
.unwrap();
let req = build_request(input, Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let fs = crate::storage::ecfs::FS::new();
let before = s3_op_total(S3Operation::PutBucketPolicy);
let err = usecase.execute_put_bucket_policy(req).await.unwrap_err();
let err = crate::app::storage_api::s3::S3::put_bucket_policy(&fs, req)
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert_eq!(s3_op_total(S3Operation::PutBucketPolicy), before + 1);
}
@@ -15,100 +15,97 @@
//! Bucket public access block use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{
DeletePublicAccessBlockInput, DeletePublicAccessBlockOutput, GetPublicAccessBlockInput, PutPublicAccessBlockInput,
PutPublicAccessBlockOutput,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
impl DefaultBucketUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_delete_public_access_block(
&self,
req: S3Request<DeletePublicAccessBlockInput>,
) -> S3Result<S3Response<DeletePublicAccessBlockOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeletePublicAccessBlockInput { bucket, .. } = req.input;
#[instrument(level = "debug", skip(self, envelope))]
pub(crate) async fn execute_delete_public_access_block(&self, envelope: RequestEnvelope, bucket: &str) -> S3Result<()> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::DeletePublicAccessBlock,
StatusCode::NO_CONTENT,
|envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
delete_bucket_config_for_incarnation(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "delete public access block", request_context, false).await;
Ok(S3Response::with_status(DeletePublicAccessBlockOutput::default(), StatusCode::NO_CONTENT))
delete_bucket_config_for_incarnation(bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.to_owned(), "delete public access block", request_context, false).await;
Ok(())
},
)
.await
}
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_get_public_access_block(
#[instrument(level = "debug", skip(self))]
pub(crate) async fn execute_get_public_access_block(
&self,
req: S3Request<GetPublicAccessBlockInput>,
bucket: &str,
) -> S3Result<PersistedPublicAccessBlockConfiguration> {
let GetPublicAccessBlockInput { bucket, .. } = req.input;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let config = match metadata_sys::get_public_access_block_config(&bucket).await {
Ok((config, _)) => config,
Err(err) => {
if err == StorageError::ConfigNotFound {
return Err(S3Error::with_message(
S3ErrorCode::Custom("NoSuchPublicAccessBlockConfiguration".into()),
"Public access block configuration does not exist".to_string(),
));
}
return Err(ApiError::from(err).into());
match metadata_sys::get_public_access_block_config(bucket).await {
Ok((config, _)) => Ok(config),
Err(StorageError::ConfigNotFound) => {
let mut error = S3Error::with_message(
S3ErrorCode::Custom("NoSuchPublicAccessBlockConfiguration".into()),
"Public access block configuration does not exist",
);
// The legacy edge used the custom code with the server-error status.
error.set_status_code(StatusCode::INTERNAL_SERVER_ERROR);
Err(error)
}
};
Ok(config)
Err(err) => Err(ApiError::from(err).into()),
}
}
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_put_public_access_block(
#[instrument(level = "debug", skip(self, envelope, prepared))]
pub(crate) async fn execute_put_public_access_block(
&self,
req: S3Request<PutPublicAccessBlockInput>,
envelope: RequestEnvelope,
bucket: &str,
prepared: S3Result<PersistedPublicAccessBlockConfiguration>,
) -> S3Result<S3Response<PutPublicAccessBlockOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let PutPublicAccessBlockInput { bucket, .. } = req.input;
) -> S3Result<()> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::PutPublicAccessBlock,
StatusCode::OK,
|envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let public_access_block_configuration = prepared?;
let data = serialize_config(&public_access_block_configuration)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "put public access block", request_context, false).await;
Ok(S3Response::new(PutPublicAccessBlockOutput::default()))
let data = serialize_config(&prepared?)?;
update_bucket_config_for_incarnation(bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.to_owned(), "put public access block", request_context, false).await;
Ok(())
},
)
.await
}
}
@@ -116,50 +113,193 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::storage_api::s3::dto::PublicAccessBlockConfiguration;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::bucket_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::owned_parity::authorize_bucket_config_mutation;
use http::Method;
use rustfs_credentials::Credentials;
use rustfs_policy::policy::action::{Action, S3Action};
use serial_test::serial;
use std::sync::Mutex;
const ORIGINAL_XML: &[u8] =
b"<PublicAccessBlockConfiguration><BlockPublicPolicy>true</BlockPublicPolicy></PublicAccessBlockConfiguration>";
async fn public_access_block_mutation(put: bool, stale: bool) {
let (method, action, operation, reload, status) = if put {
(
Method::PUT,
S3Action::PutBucketPublicAccessBlockAction,
S3Operation::PutPublicAccessBlock,
"put public access block",
200,
)
} else {
(
Method::DELETE,
S3Action::DeleteBucketPublicAccessBlockAction,
S3Operation::DeletePublicAccessBlock,
"delete public access block",
204,
)
};
let context = shared_gating_ambient().await;
let store = context.object_store();
let bucket = format!("public-access-block-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(&context)));
let mut request = build_request((), method);
request.uri = format!("/{bucket}?publicAccessBlock").parse().expect("request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "public-access-block-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
request.extensions.insert(request_context::RequestContext {
request_id: "public-access-block-fixture".to_owned(),
x_amz_request_id: "public-access-block-fixture".to_owned(),
trace_id: None,
span_id: None,
start_time: std::time::Instant::now(),
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(&server, &mut envelope, &bucket, Action::S3Action(action))
.await
.expect("bucket owner authorization installs the incarnation snapshot");
if stale {
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete authorized bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
}
metadata_sys::update(&bucket, BUCKET_PUBLIC_ACCESS_BLOCK_CONFIG, ORIGINAL_XML.to_vec())
.await
.expect("seed config");
let usecase = DefaultBucketUsecase::with_context(Some(context));
let audits = MutationCompletionProbe::install(&bucket);
let reloads = Arc::new(Mutex::new(Vec::new()));
let record = Arc::clone(&reloads);
let result = BUCKET_METADATA_RELOAD_PROBE
.scope(
(
bucket.clone(),
Box::new(move |completed, label, maintenance| {
record.lock().expect("reload observer").push((completed, label, maintenance));
}),
),
async {
if put {
let config = PersistedPublicAccessBlockConfiguration {
block_public_policy: Some(false),
..Default::default()
};
usecase.execute_put_public_access_block(envelope, &bucket, Ok(config)).await
} else {
usecase.execute_delete_public_access_block(envelope, &bucket).await
}
},
)
.await;
let stored = metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(&bucket)
.await
.expect("persisted bucket metadata")
.public_access_block_config_xml;
let reloads = reloads.lock().expect("recorded reloads");
if stale {
assert_eq!(
result.expect_err("stale authorization must not mutate replacement").code(),
&S3ErrorCode::NoSuchBucket
);
assert_eq!(stored, ORIGINAL_XML);
assert!(reloads.is_empty());
} else {
result.expect("authorized mutation");
if put {
assert_eq!(stored, b"<PublicAccessBlockConfiguration><BlockPublicPolicy>false</BlockPublicPolicy></PublicAccessBlockConfiguration>");
} else {
assert!(stored.is_empty());
}
assert_eq!(
*reloads,
[(false, reload, false), (true, reload, false)],
"metadata reload completes without scanner maintenance"
);
let entries = audits.audits.lock().expect("audit entries");
assert_eq!(entries.len(), 1);
assert_eq!(entries[0].api.name.as_deref(), Some(operation.as_str()));
assert_eq!(entries[0].api.status_code, Some(status));
assert!(audits.events.lock().expect("event entries").is_empty());
}
}
#[test]
#[serial]
fn put_public_access_block_rejects_stale_authorization() {
run_large_stack_test("public-access-block-put-stale", || public_access_block_mutation(true, true));
}
#[test]
#[serial]
fn delete_public_access_block_rejects_stale_authorization() {
run_large_stack_test("public-access-block-delete-stale", || public_access_block_mutation(false, true));
}
#[test]
#[serial]
fn put_public_access_block_reloads_and_audits() {
run_large_stack_test("public-access-block-put-completion", || public_access_block_mutation(true, false));
}
#[test]
#[serial]
fn delete_public_access_block_reloads_and_audits() {
run_large_stack_test("public-access-block-delete-completion", || public_access_block_mutation(false, false));
}
#[tokio::test]
async fn execute_delete_public_access_block_returns_internal_error_when_store_uninitialized() {
let input = DeletePublicAccessBlockInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::DELETE);
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::DELETE));
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_delete_public_access_block(req).await.unwrap_err();
let err = usecase
.execute_delete_public_access_block(envelope, "test-bucket")
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
}
#[tokio::test]
async fn execute_get_public_access_block_returns_internal_error_when_store_uninitialized() {
let input = GetPublicAccessBlockInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_get_public_access_block(req).await.unwrap_err();
let err = usecase.execute_get_public_access_block("test-bucket").await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
}
#[tokio::test]
async fn execute_put_public_access_block_returns_internal_error_when_store_uninitialized() {
let input = PutPublicAccessBlockInput::builder()
.bucket("test-bucket".to_string())
.public_access_block_configuration(PublicAccessBlockConfiguration::default())
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::PUT));
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_public_access_block(req, Ok(PersistedPublicAccessBlockConfiguration::default()))
.execute_put_public_access_block(envelope, "test-bucket", Ok(PersistedPublicAccessBlockConfiguration::default()))
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
+418 -76
View File
@@ -15,11 +15,8 @@
//! Bucket replication configuration use-case operations.
use super::*;
use crate::app::storage_api::s3::dto::{
DeleteBucketReplicationInput, DeleteBucketReplicationOutput, PutBucketReplicationInput, PutBucketReplicationOutput,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use rustfs_gateway_types::persistence::parse_replication_dto;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
fn validate_replication_config_targets(targets: &BucketTargets, config: &PersistedReplicationConfiguration) -> S3Result<()> {
let configured_arns = targets
@@ -168,6 +165,15 @@ async fn write_replication_targets_after_config_delete(
removed: usize,
expected_incarnation_id: Option<uuid::Uuid>,
) -> S3Result<()> {
#[cfg(test)]
if let Some(fail) = tests::TARGET_CLEANUP_FAILURE
.try_with(std::cell::RefCell::take)
.ok()
.flatten()
{
fail.await?;
}
let json_targets = serde_json::to_vec(&targets).map_err(to_internal_error)?;
update_bucket_config_for_incarnation(bucket, BUCKET_TARGETS_FILE, json_targets, expected_incarnation_id)
.await
@@ -210,38 +216,56 @@ async fn restore_replication_config_after_target_cleanup_failure(
}
impl DefaultBucketUsecase {
/// `site_peers` is the set of remote site-replication peer deployment ids
/// (empty when site replication is disabled). The interface layer reads it
/// from the persisted state and fails closed on a read error, so this
/// usecase stays a pure function of its inputs (layer rule: app never
/// imports interface).
pub async fn execute_delete_bucket_replication(
/// Removes operator rules and targets unique to those rules.
///
/// Preparation runs inside the audit scope and resolves the peer contract
/// before any bucket lookup or mutation.
pub(crate) async fn execute_delete_bucket_replication(
&self,
req: S3Request<DeleteBucketReplicationInput>,
envelope: RequestEnvelope,
bucket: &str,
prepare: impl Future<Output = S3Result<(HashSet<String>, OperatorRuleContract)>> + Send,
) -> S3Result<()> {
// Preserve the success status recorded by the legacy audit for an empty response.
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::DeleteBucketReplication,
StatusCode::OK,
|envelope| async move {
let (site_peers, contract) = prepare.await?;
self.delete_bucket_replication(envelope, bucket, site_peers, contract).await
},
)
.await
}
async fn delete_bucket_replication(
&self,
envelope: RequestEnvelope,
bucket: &str,
site_peers: HashSet<String>,
contract: OperatorRuleContract,
) -> S3Result<S3Response<DeleteBucketReplicationOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeleteBucketReplicationInput { bucket, .. } = req.input;
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let targets_guard = lock_bucket_targets_metadata(&bucket).await;
let replication_config = match metadata_sys::get_replication_config(&bucket).await {
let targets_guard = lock_bucket_targets_metadata(bucket).await;
let replication_config = match metadata_sys::get_replication_config(bucket).await {
Ok((config, _)) => Some(config),
Err(StorageError::ConfigNotFound) => None,
Err(err) => return Err(ApiError::from(err).into()),
};
let (remaining_config, updated_targets) = if let Some(config) = replication_config.as_ref() {
let (remaining, removable_arns) = split_replication_config_for_user_delete(config.clone(), &site_peers, contract);
let targets = replication_targets_without_arns(&bucket, &removable_arns).await?;
let targets = replication_targets_without_arns(bucket, &removable_arns).await?;
(remaining, targets)
} else {
(None, None)
@@ -252,23 +276,23 @@ impl DefaultBucketUsecase {
// S3 delete (issue #1948); only the operator-authored rules go.
Some(remaining) => {
let data = serialize_config(&remaining)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
update_bucket_config_for_incarnation(bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
}
None => {
delete_bucket_config_for_incarnation(&bucket, BUCKET_REPLICATION_CONFIG, expected_incarnation_id)
delete_bucket_config_for_incarnation(bucket, BUCKET_REPLICATION_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
}
}
if let Some((targets, removed)) = updated_targets
&& let Err(err) =
write_replication_targets_after_config_delete(&bucket, &targets, removed, expected_incarnation_id).await
write_replication_targets_after_config_delete(bucket, &targets, removed, expected_incarnation_id).await
{
if let Some(config) = replication_config.as_ref() {
return Err(restore_replication_config_after_target_cleanup_failure(
&bucket,
bucket,
config,
err,
expected_incarnation_id,
@@ -279,16 +303,16 @@ impl DefaultBucketUsecase {
}
drop(targets_guard);
notify_bucket_metadata_reload(bucket.clone(), "delete bucket replication", request_context, true).await;
notify_bucket_metadata_reload(bucket.to_owned(), "delete bucket replication", request_context, true).await;
let item = sr_bucket_meta_item(bucket.clone(), "replication-config");
let item = sr_bucket_meta_item(bucket.to_owned(), "replication-config");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket replication-config delete hook failed");
}
info!(bucket = %bucket, "deleted bucket replication config");
Ok(S3Response::new(DeleteBucketReplicationOutput::default()))
Ok(())
}
/// The stored replication configuration as the gateway's answer holds it.
@@ -335,36 +359,52 @@ impl DefaultBucketUsecase {
})
}
/// See [`Self::execute_delete_bucket_replication`] for `site_peers` and
/// `contract`. `replication_configuration` is the request body in the
/// engine's shape: the s3s edge (`storage::ecfs`) converts it, and this use
/// case never reads `req.input.replication_configuration`.
pub async fn execute_put_bucket_replication(
/// Replaces operator rules after the interface resolves the peer contract.
pub(crate) async fn execute_put_bucket_replication(
&self,
req: S3Request<PutBucketReplicationInput>,
envelope: RequestEnvelope,
bucket: &str,
prepare: impl Future<Output = S3Result<(PersistedReplicationConfiguration, HashSet<String>, OperatorRuleContract)>> + Send,
) -> S3Result<()> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::PutBucketReplication,
StatusCode::OK,
|envelope| async move {
let (replication_configuration, site_peers, contract) = prepare.await?;
self.put_bucket_replication(envelope, bucket, replication_configuration, site_peers, contract)
.await
},
)
.await
}
async fn put_bucket_replication(
&self,
envelope: RequestEnvelope,
bucket: &str,
replication_configuration: PersistedReplicationConfiguration,
site_peers: HashSet<String>,
contract: OperatorRuleContract,
) -> S3Result<S3Response<PutBucketReplicationOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let PutBucketReplicationInput { bucket, .. } = req.input;
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
info!(bucket = %bucket, "updating bucket replication config");
validate_replication_config_capabilities(&replication_configuration)?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let targets_guard = lock_bucket_targets_metadata(&bucket).await;
validate_bucket_replication_update(&bucket, &replication_configuration).await?;
let existing_config = match metadata_sys::get_replication_config(&bucket).await {
let targets_guard = lock_bucket_targets_metadata(bucket).await;
validate_bucket_replication_update(bucket, &replication_configuration).await?;
let existing_config = match metadata_sys::get_replication_config(bucket).await {
Ok((config, _)) => Some(config),
Err(StorageError::ConfigNotFound) => None,
Err(err) => return Err(ApiError::from(err).into()),
@@ -372,14 +412,14 @@ impl DefaultBucketUsecase {
let replication_configuration =
merge_user_replication_config_update(replication_configuration, existing_config, &site_peers, contract);
let data = serialize_config(&replication_configuration)?;
update_bucket_config_for_incarnation(&bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
update_bucket_config_for_incarnation(bucket, BUCKET_REPLICATION_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
drop(targets_guard);
notify_bucket_metadata_reload(bucket.clone(), "put bucket replication", request_context, true).await;
notify_bucket_metadata_reload(bucket.to_owned(), "put bucket replication", request_context, true).await;
let mut item = sr_bucket_meta_item(bucket.clone(), "replication-config");
let mut item = sr_bucket_meta_item(bucket.to_owned(), "replication-config");
item.replication_config = Some(
serialize_config(&replication_configuration).and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?,
);
@@ -387,19 +427,29 @@ impl DefaultBucketUsecase {
warn!(bucket = %bucket, error = ?err, "site replication bucket replication-config hook failed");
}
Ok(S3Response::new(PutBucketReplicationOutput::default()))
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::bucket_usecase::test_support::{build_request, owned_envelope};
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::owned_parity::{ReqInfo, authorize_bucket_config_mutation};
use crate::site_replication::hooks::{BUCKET_META_HOOK_CALLS, BucketMetaHookCall};
use futures::{FutureExt, future::BoxFuture};
use http::Method;
use rustfs_credentials::Credentials;
use rustfs_gateway_types::dto::Status;
use rustfs_gateway_types::persistence::{
PersistedEncryptionConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
};
use rustfs_policy::policy::action::{Action, S3Action};
use serial_test::serial;
use std::cell::RefCell;
use std::sync::Mutex;
fn replication_rule_for_target(arn: &str) -> PersistedReplicationRule {
PersistedReplicationRule {
@@ -834,16 +884,11 @@ mod tests {
#[tokio::test]
async fn execute_delete_bucket_replication_returns_internal_error_when_store_uninitialized() {
let input = DeleteBucketReplicationInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::DELETE);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_delete_bucket_replication(req, HashSet::new(), OperatorRuleContract::Derived)
.execute_delete_bucket_replication(owned_envelope(Method::DELETE), "test-bucket", async {
Ok((HashSet::new(), OperatorRuleContract::Derived))
})
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
@@ -869,17 +914,11 @@ mod tests {
role: "arn:aws:iam::123456789012:role/test".to_string(),
rules: vec![replication_rule_for_target("arn:rustfs:replication:us-east-1:target:bucket")],
};
let input = PutBucketReplicationInput::builder()
.bucket("test-bucket".to_string())
.replication_configuration(Default::default())
.build()
.unwrap();
let req = build_request(input, Method::PUT);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_replication(req, configuration, HashSet::new(), OperatorRuleContract::Derived)
.execute_put_bucket_replication(owned_envelope(Method::PUT), "test-bucket", async {
Ok((configuration, HashSet::new(), OperatorRuleContract::Derived))
})
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
@@ -893,19 +932,10 @@ mod tests {
role: String::new(),
rules: vec![rule],
};
let input = PutBucketReplicationInput::builder()
.bucket("test-bucket".to_string())
.replication_configuration(Default::default())
.build()
.unwrap();
let err = DefaultBucketUsecase::without_context()
.execute_put_bucket_replication(
build_request(input, Method::PUT),
configuration,
HashSet::new(),
OperatorRuleContract::Derived,
)
.execute_put_bucket_replication(owned_envelope(Method::PUT), "test-bucket", async {
Ok((configuration, HashSet::new(), OperatorRuleContract::Derived))
})
.await
.expect_err("unsupported fields must be rejected before store access");
@@ -915,4 +945,316 @@ mod tests {
Some("replication field Destination.Account is not supported by this RustFS version")
);
}
const PEER_ARN: &str = "arn:rustfs:replication::peer-dep:bucket";
const USER_ARN: &str = "arn:rustfs:replication:us-east-1:user:bucket";
const SITE_RULE: &str = "site-repl-peer-dep";
tokio::task_local! {
pub(super) static TARGET_CLEANUP_FAILURE: RefCell<Option<BoxFuture<'static, S3Result<()>>>>;
}
fn put_configuration() -> PersistedReplicationConfiguration {
PersistedReplicationConfiguration {
role: String::new(),
rules: vec![
replication_rule_with_id(USER_ARN, "new-operator", 3),
replication_rule_with_id(PEER_ARN, "shared-peer-target", 4),
],
}
}
async fn seed_replication(store: &ECStore, bucket: &str, operator: &str) -> (Vec<u8>, Vec<u8>) {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket for replication");
metadata_sys::update(
bucket,
BUCKET_VERSIONING_CONFIG,
b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec(),
)
.await
.expect("enable versioning");
let config = PersistedReplicationConfiguration {
role: String::new(),
rules: vec![
replication_rule_with_id(USER_ARN, operator, 1),
replication_rule_with_id(PEER_ARN, SITE_RULE, 2),
],
};
let targets = BucketTargets {
targets: [PEER_ARN, USER_ARN]
.into_iter()
.map(|arn| BucketTarget {
source_bucket: bucket.to_owned(),
arn: arn.to_owned(),
target_type: BucketTargetType::ReplicationService,
..Default::default()
})
.collect(),
};
let bytes = serialize(&config).expect("seed configuration bytes");
let targets = serde_json::to_vec(&targets).expect("seed target bytes");
metadata_sys::update(bucket, BUCKET_TARGETS_FILE, targets.clone())
.await
.expect("seed target metadata");
metadata_sys::update(bucket, BUCKET_REPLICATION_CONFIG, bytes.clone())
.await
.expect("seed replication configuration");
(bytes, targets)
}
async fn persisted_replication(bucket: &str) -> (Vec<u8>, Vec<u8>) {
let stored = metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(bucket)
.await
.expect("disk metadata");
(stored.replication_config_xml, stored.bucket_targets_config_json)
}
async fn authorize_replication(context: &Arc<AppContext>, bucket: &str, method: Method) -> RequestEnvelope {
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(context)));
let mut request = build_request((), method);
request.uri = format!("/{bucket}?replication").parse().expect("replication URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "replication-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(
&server,
&mut envelope,
bucket,
Action::S3Action(S3Action::PutReplicationConfigurationAction),
)
.await
.expect("snapshot authorized bucket");
envelope
}
async fn edit_replication(
usecase: &DefaultBucketUsecase,
envelope: RequestEnvelope,
bucket: &str,
put: bool,
) -> S3Result<()> {
let peers = HashSet::from(["peer-dep".to_owned()]);
if put {
usecase
.execute_put_bucket_replication(envelope, bucket, async {
Ok((put_configuration(), peers, OperatorRuleContract::Derived))
})
.await
} else {
usecase
.execute_delete_bucket_replication(envelope, bucket, async { Ok((peers, OperatorRuleContract::Derived)) })
.await
}
}
#[derive(Debug, PartialEq, Eq)]
struct Reload {
completed: bool,
maintenance: bool,
target_lock_available: bool,
hook_calls: usize,
}
async fn observe_replication_edit(
bucket: &str,
edit: impl Future<Output = S3Result<()>>,
) -> (S3Result<()>, Vec<Reload>, Vec<BucketMetaHookCall>) {
let reloads = Arc::new(Mutex::new(Vec::new()));
let record = Arc::clone(&reloads);
let target_bucket = bucket.to_owned();
let (result, hooks) = BUCKET_META_HOOK_CALLS
.scope(
RefCell::default(),
BUCKET_METADATA_RELOAD_PROBE.scope(
(
bucket.to_owned(),
Box::new(move |completed, _, maintenance| {
record.lock().expect("reloads").push(Reload {
completed,
maintenance,
target_lock_available: lock_bucket_targets_metadata(&target_bucket).now_or_never().is_some(),
hook_calls: BUCKET_META_HOOK_CALLS.with(|calls| calls.borrow().len()),
});
}),
),
async {
let result = edit.await;
(result, BUCKET_META_HOOK_CALLS.with(RefCell::take))
},
),
)
.await;
let reloads = std::mem::take(&mut *reloads.lock().expect("reloads"));
(result, reloads, hooks)
}
#[test]
#[serial]
fn owned_replication_preserves_peer_rules_and_finishes_reload_before_site_hook() {
run_large_stack_test("owned-replication-peer-rules", || async {
let context = shared_gating_ambient().await;
let bucket = format!("replication-peer-{}", uuid::Uuid::new_v4().simple());
let (original, _) = seed_replication(&context.object_store(), &bucket, "old-operator").await;
let usecase = DefaultBucketUsecase::with_context(Some(Arc::clone(&context)));
for (put, method) in [(true, Method::PUT), (false, Method::DELETE)] {
let envelope = authorize_replication(&context, &bucket, method).await;
let (result, reloads, hooks) =
observe_replication_edit(&bucket, edit_replication(&usecase, envelope, &bucket, put)).await;
result.expect("replication edit");
assert_eq!(
reloads,
[
Reload {
completed: false,
maintenance: true,
target_lock_available: true,
hook_calls: 0
},
Reload {
completed: true,
maintenance: true,
target_lock_available: true,
hook_calls: 0
},
]
);
let (xml, target_bytes) = persisted_replication(&bucket).await;
assert_ne!(xml, original);
let config = rustfs_gateway_types::persistence::parse_replication(&xml).expect("stored XML");
let ids: HashSet<_> = config.rules.iter().filter_map(|rule| rule.id.as_deref()).collect();
let targets: BucketTargets = serde_json::from_slice(&target_bytes).expect("stored targets");
let arns: HashSet<_> = targets.targets.iter().map(|target| target.arn.as_str()).collect();
if put {
assert_eq!(ids, HashSet::from([SITE_RULE, "new-operator", "shared-peer-target"]));
assert_eq!(arns, HashSet::from([PEER_ARN, USER_ARN]));
} else {
assert_eq!(ids, HashSet::from([SITE_RULE]));
assert_eq!(arns, HashSet::from([PEER_ARN]));
}
assert_eq!(hooks.len(), 1);
assert_eq!(
(hooks[0].bucket.as_str(), hooks[0].item_type.as_str()),
(bucket.as_str(), "replication-config")
);
assert_eq!(hooks[0].replication_config.as_deref().map(str::as_bytes), put.then_some(xml.as_slice()));
}
});
}
#[test]
#[serial]
fn owned_replication_stale_edits_leave_replacement_bytes_and_skip_callbacks() {
run_large_stack_test("owned-replication-stale", || async {
let context = shared_gating_ambient().await;
let store = context.object_store();
let usecase = DefaultBucketUsecase::with_context(Some(Arc::clone(&context)));
for (method, preserve_peer) in [(Method::PUT, true), (Method::DELETE, true), (Method::DELETE, false)] {
let bucket = format!("replication-stale-{}", uuid::Uuid::new_v4().simple());
seed_replication(&store, &bucket, "old-operator").await;
let put = method == Method::PUT;
let envelope = authorize_replication(&context, &bucket, method).await;
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete authorized bucket");
let replacement = seed_replication(&store, &bucket, "replacement").await;
let edit = async {
if preserve_peer {
edit_replication(&usecase, envelope, &bucket, put).await
} else {
usecase
.execute_delete_bucket_replication(envelope, &bucket, async {
Ok((HashSet::new(), OperatorRuleContract::Derived))
})
.await
}
};
let (result, reloads, hooks) = observe_replication_edit(&bucket, edit).await;
assert_eq!(
(
result.err().map(|err| err.code().clone()),
persisted_replication(&bucket).await,
reloads.len(),
hooks.len()
),
(Some(S3ErrorCode::NoSuchBucket), replacement, 0, 0),
"stale {}",
if put { "Put" } else { "Delete" }
);
}
});
}
#[test]
#[serial]
fn owned_replication_cleanup_error_restores_original_and_never_overwrites_a_replacement() {
run_large_stack_test("owned-replication-cleanup", || async {
let context = shared_gating_ambient().await;
let store = context.object_store();
let usecase = DefaultBucketUsecase::with_context(Some(Arc::clone(&context)));
for (replace, fail_target_write) in [(false, true), (true, true), (true, false)] {
let bucket = format!("replication-cleanup-{}", uuid::Uuid::new_v4().simple());
let original = seed_replication(&store, &bucket, "old-operator").await;
let envelope = authorize_replication(&context, &bucket, Method::DELETE).await;
let expected_storage = Arc::new(Mutex::new(original));
let output = Arc::clone(&expected_storage);
let replacement_bucket = bucket.clone();
let old_store = Arc::clone(&store);
let fail = Box::pin(async move {
let (after_delete, _) = persisted_replication(&replacement_bucket).await;
assert!(!String::from_utf8_lossy(&after_delete).contains("old-operator"));
assert!(lock_bucket_targets_metadata(&replacement_bucket).now_or_never().is_none());
if replace {
old_store
.delete_bucket(&replacement_bucket, &DeleteBucketOptions::default())
.await
.expect("delete original before restoration");
let replacement = seed_replication(&old_store, &replacement_bucket, "replacement").await;
*output.lock().expect("replacement bytes") = replacement;
}
if fail_target_write {
Err(s3_error!(InternalError, "replication target cleanup failed"))
} else {
Ok(())
}
});
let (result, reloads, hooks) = TARGET_CLEANUP_FAILURE
.scope(
RefCell::new(Some(fail)),
observe_replication_edit(&bucket, edit_replication(&usecase, envelope, &bucket, false)),
)
.await;
let expected = if fail_target_write {
S3ErrorCode::InternalError
} else {
S3ErrorCode::NoSuchBucket
};
let stored = persisted_replication(&bucket).await;
assert_eq!(
(result.as_ref().err().map(|error| error.code().clone()), stored),
(Some(expected), expected_storage.lock().expect("expected bytes").clone())
);
if fail_target_write {
assert_eq!(result.unwrap_err().message(), Some("replication target cleanup failed"));
}
assert!(reloads.is_empty() && hooks.is_empty());
}
});
}
}
@@ -12,4 +12,46 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bucket request-payment configuration use-case operations (reserved for the next batch).
//! Bucket request-payment configuration shared by the S3 edges.
use super::BUCKET_REQUEST_PAYMENT_CONFIG;
use super::{
ApiError, DefaultBucketUsecase, StorageError, bucket_config_mutation_incarnation, metadata_sys, serialize,
update_bucket_config_for_incarnation,
};
use crate::app::RequestEnvelope;
use rustfs_gateway_types::persistence::PersistedRequestPaymentConfiguration;
use rustfs_s3_types::S3Result;
impl DefaultBucketUsecase {
pub(crate) async fn execute_get_bucket_request_payment(
&self,
bucket: &str,
) -> S3Result<PersistedRequestPaymentConfiguration> {
self.ensure_config_bucket(bucket).await?;
match metadata_sys::get_request_payment_config(bucket).await {
Ok((config, _)) => Ok(config),
Err(StorageError::ConfigNotFound) => Ok(PersistedRequestPaymentConfiguration {
payer: "BucketOwner".to_owned(),
}),
Err(err) => Err(ApiError::from(err).into()),
}
}
pub(crate) async fn execute_put_bucket_request_payment(
&self,
envelope: RequestEnvelope,
bucket: &str,
configuration: impl FnOnce() -> S3Result<PersistedRequestPaymentConfiguration> + Send,
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
self.ensure_config_bucket(bucket).await?;
let config = configuration()?;
let data = serialize(&config).map_err(|err| rustfs_s3_types::s3_error!(MalformedXML, "{err}"))?;
update_bucket_config_for_incarnation(bucket, BUCKET_REQUEST_PAYMENT_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
Ok(())
}
}
+98 -81
View File
@@ -15,58 +15,63 @@
//! Bucket tagging use-case operations.
use super::*;
use crate::app::storage_api::s3::dto;
use crate::app::storage_api::s3::dto::{
DeleteBucketTaggingInput, DeleteBucketTaggingOutput, GetBucketTaggingInput, PutBucketTaggingInput, PutBucketTaggingOutput,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use crate::app::storage_api::bucket_usecase::bucket::metadata::{MAX_BUCKET_TAGS, validate_tag_set_entries};
use rustfs_s3_types::s3_error;
impl DefaultBucketUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_delete_bucket_tagging(
#[instrument(level = "debug", skip(self, envelope))]
pub(crate) async fn execute_delete_bucket_tagging(
&self,
req: S3Request<DeleteBucketTaggingInput>,
) -> S3Result<S3Response<DeleteBucketTaggingOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let DeleteBucketTaggingInput { bucket, .. } = req.input;
envelope: RequestEnvelope,
bucket: &str,
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::DeleteBucketTagging, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
metadata_sys::update_s3_tagging_config(&bucket, None, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
metadata_sys::update_s3_tagging_config(bucket, None, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "delete bucket tagging", request_context, false).await;
notify_bucket_metadata_reload(bucket.to_owned(), "delete bucket tagging", request_context, false).await;
let item = sr_bucket_meta_item(bucket.clone(), "tags");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket tagging delete hook failed");
}
let item = sr_bucket_meta_item(bucket.to_owned(), "tags");
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket tagging delete hook failed");
}
rustfs_scanner::record_dirty_usage_bucket_from_producer(
&bucket,
rustfs_scanner::SegmentInvalidationProducerIdentity::BucketMetadata,
);
Ok(S3Response::new(DeleteBucketTaggingOutput {}))
rustfs_scanner::record_dirty_usage_bucket_from_producer(
bucket,
rustfs_scanner::SegmentInvalidationProducerIdentity::BucketMetadata,
);
Ok(())
})
.await
}
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_get_bucket_tagging(&self, req: S3Request<GetBucketTaggingInput>) -> S3Result<PersistedTagging> {
let GetBucketTaggingInput { bucket, .. } = req.input;
/// Returns the stored bucket tagging configuration after checking the bucket exists.
///
/// # Errors
///
/// Returns `InternalError` if the store is uninitialized, `NoSuchTagSet`
/// if the configuration is absent, or the storage error from either lookup.
#[instrument(level = "debug", skip(self, bucket))]
pub async fn execute_get_bucket_tagging(&self, bucket: &str) -> rustfs_s3_types::S3Result<PersistedTagging> {
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
store
.get_bucket_info(&bucket, &BucketOptions::default())
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let tagging = match metadata_sys::get_tagging_config(&bucket).await {
let tagging = match metadata_sys::get_tagging_config(bucket).await {
Ok((tags, _)) => tags,
Err(err) => {
if err == StorageError::ConfigNotFound {
return Err(S3Error::with_message(S3ErrorCode::NoSuchTagSet, "The TagSet does not exist".to_string()));
return Err(rustfs_s3_types::s3_error!(NoSuchTagSet, "The TagSet does not exist"));
}
warn!(
component = LOG_COMPONENT_APP,
@@ -83,51 +88,50 @@ impl DefaultBucketUsecase {
Ok(tagging)
}
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_put_bucket_tagging(
#[instrument(level = "debug", skip(self, envelope, tagging))]
pub(crate) async fn execute_put_bucket_tagging(
&self,
req: S3Request<PutBucketTaggingInput>,
envelope: RequestEnvelope,
bucket: &str,
tagging: PersistedTagging,
) -> S3Result<S3Response<PutBucketTaggingOutput>> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&req, &req.input.bucket)?;
let request_context = req.extensions.get::<request_context::RequestContext>().cloned();
let PutBucketTaggingInput { bucket, .. } = req.input;
let s3_tag_set: Vec<dto::Tag> = tagging
.tag_set
.iter()
.map(|tag| dto::Tag {
key: tag.key.clone(),
value: tag.value.clone(),
})
.collect();
crate::app::storage_api::bucket_usecase::s3_api::tagging::validate_bucket_tag_set(&s3_tag_set)?;
) -> rustfs_s3_types::S3Result<()> {
RequestEnvelope::audit_only_operation(envelope, S3Operation::PutBucketTagging, StatusCode::OK, |envelope| async move {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
let request_context = envelope.request_context().cloned();
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
if tagging.tag_set.len() > MAX_BUCKET_TAGS {
return Err(s3_error!(InvalidTag, "Cannot have more than 50 tags per bucket"));
}
validate_tag_set_entries(&tagging.tag_set).map_err(|err| s3_error!(InvalidTag, "{err}"))?;
store
.get_bucket_info(&bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
let Some(store) = self.object_store() else {
return Err(s3_error!(InternalError, "Not init"));
};
metadata_sys::update_s3_tagging_config(&bucket, Some(tagging.clone()), expected_incarnation_id)
.await
.map_err(ApiError::from)?;
store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
notify_bucket_metadata_reload(bucket.clone(), "put bucket tagging", request_context, false).await;
metadata_sys::update_s3_tagging_config(bucket, Some(tagging.clone()), expected_incarnation_id)
.await
.map_err(ApiError::from)?;
let mut item = sr_bucket_meta_item(bucket.clone(), "tags");
item.tags = Some(serialize_config(&tagging).and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket tagging hook failed");
}
notify_bucket_metadata_reload(bucket.to_owned(), "put bucket tagging", request_context, false).await;
rustfs_scanner::record_dirty_usage_bucket_from_producer(
&bucket,
rustfs_scanner::SegmentInvalidationProducerIdentity::BucketMetadata,
);
Ok(S3Response::new(PutBucketTaggingOutput::default()))
let mut item = sr_bucket_meta_item(bucket.to_owned(), "tags");
item.tags = Some(serialize_config(&tagging).and_then(|bytes| String::from_utf8(bytes).map_err(to_internal_error))?);
if let Err(err) = site_replication_bucket_meta_hook(item).await {
warn!(bucket = %bucket, error = ?err, "site replication bucket tagging hook failed");
}
rustfs_scanner::record_dirty_usage_bucket_from_producer(
bucket,
rustfs_scanner::SegmentInvalidationProducerIdentity::BucketMetadata,
);
Ok(())
})
.await
}
}
@@ -135,21 +139,16 @@ impl DefaultBucketUsecase {
mod tests {
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::storage_api::s3::dto::{Tag, Tagging};
use crate::app::storage_api::s3::dto::{self, PutBucketTaggingInput, Tag, Tagging};
use http::Method;
use rustfs_s3_types::S3ErrorCode;
#[tokio::test]
async fn execute_get_bucket_tagging_returns_internal_error_when_store_uninitialized() {
let input = GetBucketTaggingInput::builder()
.bucket("test-bucket".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let usecase = DefaultBucketUsecase::without_context();
let err = usecase.execute_get_bucket_tagging(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let err = usecase.execute_get_bucket_tagging("test-bucket").await.unwrap_err();
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InternalError);
}
#[test]
@@ -214,7 +213,9 @@ mod tests {
req.extensions.insert(ReqInfo::default());
req.extensions.insert(slot.clone());
fs.put_bucket_tagging(&mut req).await.expect("permission allows tag updates");
let result = usecase.execute_put_bucket_tagging(req, persisted).await;
let result = usecase
.execute_put_bucket_tagging(RequestEnvelope::from_s3s(&req), &req.input.bucket, persisted)
.await;
if valid {
result.expect("valid tags are persisted");
} else {
@@ -245,9 +246,25 @@ mod tests {
let usecase = DefaultBucketUsecase::without_context();
let err = usecase
.execute_put_bucket_tagging(req, PersistedTagging::default())
.execute_put_bucket_tagging(RequestEnvelope::from_s3s(&req), &req.input.bucket, PersistedTagging::default())
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
let tag = rustfs_gateway_types::cors_tagging::PersistedTag {
key: Some("team".to_owned()),
value: Some("storage".to_owned()),
};
for (tag_set, message) in [
(vec![tag.clone(), tag.clone()], "Cannot provide multiple Tags with the same key"),
(vec![tag; MAX_BUCKET_TAGS + 1], "Cannot have more than 50 tags per bucket"),
] {
let err = usecase
.execute_put_bucket_tagging(RequestEnvelope::from_s3s(&req), &req.input.bucket, PersistedTagging { tag_set })
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidTag);
assert_eq!(err.message(), Some(message));
}
}
}
+225 -1
View File
@@ -12,4 +12,228 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Bucket website configuration use-case operations (reserved for the next batch).
//! Bucket website configuration shared by the S3 edges.
use super::BUCKET_WEBSITE_CONFIG;
use super::{
ApiError, DefaultBucketUsecase, StorageError, bucket_config_mutation_incarnation, delete_bucket_config_for_incarnation,
metadata_sys, serialize, update_bucket_config_for_incarnation,
};
use crate::app::RequestEnvelope;
use crate::app::storage_api::bucket_usecase::validate_website_configuration;
use rustfs_gateway_types::persistence::PersistedWebsiteConfiguration;
use rustfs_s3_types::S3Result;
impl DefaultBucketUsecase {
pub(crate) async fn execute_get_bucket_website(&self, bucket: &str) -> S3Result<PersistedWebsiteConfiguration> {
self.ensure_config_bucket(bucket).await?;
match metadata_sys::get_website_config(bucket).await {
Ok((config, _)) => Ok(config),
Err(StorageError::ConfigNotFound) => Err(rustfs_s3_types::s3_error!(
NoSuchWebsiteConfiguration,
"The specified bucket does not have a website configuration."
)),
Err(err) => Err(ApiError::from(err).into()),
}
}
pub(crate) async fn execute_put_bucket_website(
&self,
envelope: RequestEnvelope,
bucket: &str,
configuration: impl FnOnce() -> S3Result<PersistedWebsiteConfiguration> + Send,
) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
self.ensure_config_bucket(bucket).await?;
let config = configuration()?;
validate_website_configuration(&config)?;
let data = serialize(&config).map_err(|err| rustfs_s3_types::s3_error!(MalformedXML, "{err}"))?;
update_bucket_config_for_incarnation(bucket, BUCKET_WEBSITE_CONFIG, data, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
Ok(())
}
pub(crate) async fn execute_delete_bucket_website(&self, envelope: RequestEnvelope, bucket: &str) -> S3Result<()> {
let expected_incarnation_id = bucket_config_mutation_incarnation(&envelope, bucket)?;
self.ensure_config_bucket(bucket).await?;
delete_bucket_config_for_incarnation(bucket, BUCKET_WEBSITE_CONFIG, expected_incarnation_id)
.await
.map_err(ApiError::from)?;
Ok(())
}
}
#[cfg(test)]
mod tests {
use super::super::{BUCKET_ACCELERATE_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG};
use super::*;
use crate::app::bucket_usecase::test_support::build_request;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::owned_parity::{ReqInfo, authorize_bucket_config_mutation};
use rustfs_credentials::Credentials;
use rustfs_gateway_types::persistence::{parse_accelerate, parse_request_payment, parse_website};
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_s3_types::S3ErrorCode;
use serial_test::serial;
use std::sync::Arc;
#[test]
#[serial]
fn core_config_puts_defer_conversion_until_after_bucket_lookup() {
run_large_stack_test("core-config-lookup-before-conversion", || async {
let ambient = shared_gating_ambient().await;
let usecase = DefaultBucketUsecase::with_context(Some(ambient));
let bucket = format!("core-config-missing-{}", uuid::Uuid::new_v4().simple());
let request = build_request((), http::Method::PUT);
let mut converted = [false; 3];
let accelerate = usecase
.execute_put_bucket_accelerate_configuration(RequestEnvelope::from_s3s(&request), &bucket, || {
converted[0] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid acceleration"))
})
.await;
let payment = usecase
.execute_put_bucket_request_payment(RequestEnvelope::from_s3s(&request), &bucket, || {
converted[1] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid request payment"))
})
.await;
let website = usecase
.execute_put_bucket_website(RequestEnvelope::from_s3s(&request), &bucket, || {
converted[2] = true;
Err(rustfs_s3_types::s3_error!(MalformedXML, "invalid website"))
})
.await;
assert_eq!(
[accelerate, payment, website].map(|result| result.err().map(|error| error.code().clone())),
[
Some(S3ErrorCode::NoSuchBucket),
Some(S3ErrorCode::NoSuchBucket),
Some(S3ErrorCode::NoSuchBucket)
],
);
assert_eq!(converted, [false; 3]);
});
}
#[test]
#[serial]
fn core_config_mutations_reject_stale_incarnations() {
use super::super::{BucketOperations, DeleteBucketOptions, MakeBucketOptions};
const OLD_ACCELERATE: &[u8] = b"<AccelerateConfiguration><Status>Suspended</Status></AccelerateConfiguration>";
const NEW_ACCELERATE: &[u8] = b"<AccelerateConfiguration><Status>Enabled</Status></AccelerateConfiguration>";
const OLD_PAYMENT: &[u8] = b"<RequestPaymentConfiguration><Payer>BucketOwner</Payer></RequestPaymentConfiguration>";
const NEW_PAYMENT: &[u8] = b"<RequestPaymentConfiguration><Payer>Requester</Payer></RequestPaymentConfiguration>";
const OLD_WEBSITE: &[u8] =
b"<WebsiteConfiguration><IndexDocument><Suffix>old.html</Suffix></IndexDocument></WebsiteConfiguration>";
const NEW_WEBSITE: &[u8] =
b"<WebsiteConfiguration><IndexDocument><Suffix>new.html</Suffix></IndexDocument></WebsiteConfiguration>";
run_large_stack_test("core-config-stale-incarnation", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let server = ServerContextSlot::new();
assert!(server.install(Arc::clone(&ambient)));
let usecase = DefaultBucketUsecase::with_context(Some(Arc::clone(&ambient)));
let mut results = Vec::new();
for (method, query, file, before) in [
(http::Method::PUT, "accelerate", BUCKET_ACCELERATE_CONFIG, OLD_ACCELERATE),
(http::Method::PUT, "requestPayment", BUCKET_REQUEST_PAYMENT_CONFIG, OLD_PAYMENT),
(http::Method::PUT, "website", BUCKET_WEBSITE_CONFIG, OLD_WEBSITE),
(http::Method::DELETE, "website", BUCKET_WEBSITE_CONFIG, OLD_WEBSITE),
] {
let bucket = format!("core-config-stale-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create original bucket");
let mut request = build_request((), method.clone());
request.uri = format!("/{bucket}?{query}").parse().expect("bucket request URI");
request.extensions.insert(Arc::clone(&server));
request.extensions.insert(ReqInfo {
cred: Some(Credentials {
access_key: "core-config-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..Default::default()
});
let mut envelope = RequestEnvelope::from_s3s(&request);
authorize_bucket_config_mutation(
&server,
&mut envelope,
&bucket,
Action::S3Action(S3Action::PutBucketPolicyAction),
)
.await
.expect("owner authorization captures the original incarnation");
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create replacement bucket");
metadata_sys::update(&bucket, file, before.to_vec())
.await
.expect("seed replacement configuration");
let result = match (method, query) {
(http::Method::PUT, "accelerate") => {
usecase
.execute_put_bucket_accelerate_configuration(envelope, &bucket, || {
Ok(parse_accelerate(NEW_ACCELERATE).expect("changed accelerate document"))
})
.await
}
(http::Method::PUT, "requestPayment") => {
usecase
.execute_put_bucket_request_payment(envelope, &bucket, || {
Ok(parse_request_payment(NEW_PAYMENT).expect("changed request payment document"))
})
.await
}
(http::Method::PUT, "website") => {
usecase
.execute_put_bucket_website(envelope, &bucket, || {
Ok(parse_website(NEW_WEBSITE).expect("changed website document"))
})
.await
}
(http::Method::DELETE, "website") => usecase.execute_delete_bucket_website(envelope, &bucket).await,
_ => unreachable!("only core bucket configuration mutations"),
};
let stored = metadata_sys::get_global_bucket_metadata_sys()
.expect("initialized metadata system")
.read()
.await
.get_config_from_disk(&bucket)
.await
.expect("replacement metadata");
let after = match file {
BUCKET_ACCELERATE_CONFIG => stored.accelerate_config_xml,
BUCKET_REQUEST_PAYMENT_CONFIG => stored.request_payment_config_xml,
BUCKET_WEBSITE_CONFIG => stored.website_config_xml,
_ => unreachable!("only core bucket configurations"),
};
results.push((query, result.err().map(|err| err.code().clone()), after == before));
}
assert_eq!(
results,
["accelerate", "requestPayment", "website", "website"].map(|query| (
query,
Some(S3ErrorCode::NoSuchBucket),
true
)),
"each operation must reject the stale authorization and preserve the replacement bytes"
);
});
}
}
+6
View File
@@ -664,6 +664,9 @@ impl AuthorizationRequest for RequestEnvelope {
fn requested_object_tags(&self) -> Option<&RequestedObjectTags> {
self.requested_object_tags()
}
fn take_requested_object_tags(&mut self) -> Option<RequestedObjectTags> {
self.requested_object_tags.take()
}
fn install_requested_retention(&mut self, retention: RequestedObjectLockRetention) {
self.requested_object_lock_retention = Some(retention);
}
@@ -691,6 +694,9 @@ impl AuthorizationRequest for RequestEnvelope {
fn object_tag_conditions(&self) -> Option<&ObjectTagConditions> {
self.object_tag_conditions()
}
fn take_object_tag_conditions(&mut self) -> Option<ObjectTagConditions> {
self.object_tag_conditions.take()
}
fn install_object_tag_conditions(&mut self, conditions: ObjectTagConditions) {
self.install_object_tag_conditions(conditions);
}
+5
View File
@@ -377,6 +377,11 @@ impl AppContext {
self
}
pub(crate) fn with_test_notify_interface(mut self, notify: Arc<dyn NotifyInterface>) -> Self {
self.notify = notify;
self
}
pub(crate) fn with_test_notification_system_interface(
mut self,
notification_system: Arc<dyn NotificationSystemInterface>,
@@ -108,17 +108,6 @@ async fn persist_scanner_snapshot(ecstore: Arc<ECStore>, info: DataUsageInfo) {
rustfs_scanner::scanner::store_data_usage_in_backend(CancellationToken::new(), ecstore, receiver).await;
}
async fn bind_current_bucket_incarnations(ecstore: &ECStore, info: &mut DataUsageInfo) {
for (bucket, usage) in &mut info.buckets_usage {
let incarnation = ecstore
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("load the current incarnation for the covered snapshot");
assert!(!incarnation.is_nil(), "covered snapshot must bind a non-nil bucket identity");
usage.bucket_incarnation = Some(incarnation.to_string());
}
}
#[tokio::test]
#[serial]
async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
@@ -185,7 +174,6 @@ async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
seeded_info.bucket_sizes.insert(overlay_bucket.clone(), 0);
seeded_info.buckets_count = u64::try_from(seeded_info.buckets_usage.len()).expect("test bucket count should fit into u64");
seeded_info.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut seeded_info).await;
persist_scanner_snapshot(ecstore.clone(), seeded_info).await;
record_bucket_object_write_memory(&overlay_bucket, None, 512).await;
@@ -353,7 +341,6 @@ async fn data_usage_endpoint_treats_legacy_partial_stats_as_unknown_until_covere
completed.bucket_sizes.insert(small_bucket.clone(), 57_344);
completed.buckets_count = u64::try_from(completed.buckets_usage.len()).expect("test bucket count should fit into u64");
completed.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut completed).await;
store_data_usage_in_backend(completed, ecstore.clone())
.await
.expect("persist complete covered snapshot");
@@ -332,7 +332,7 @@ async fn delete_objects_resolves_bucket_versioning_once_per_request() {
let lookups_before = VERSIONING_CONFIG_LOOKUPS.load(Ordering::SeqCst);
let snapshots_before = DELETE_CONFIG_SNAPSHOT_LOADS.load(Ordering::SeqCst);
let response = usecase
.execute_delete_objects(req)
.execute_delete_objects_from_s3s(req)
.await
.expect("delete objects should succeed");
@@ -389,7 +389,7 @@ async fn delete_objects_treats_empty_directory_version_as_absent() {
});
let response = usecase
.execute_delete_objects(req)
.execute_delete_objects_from_s3s(req)
.await
.expect("empty VersionId should be treated as an absent version");
let deleted = response
@@ -438,7 +438,7 @@ async fn delete_objects_preserves_explicit_null_directory_version() {
});
let response = usecase
.execute_delete_objects(req)
.execute_delete_objects_from_s3s(req)
.await
.expect("explicit null VersionId should delete the directory's null version");
let deleted = response
@@ -839,7 +839,7 @@ async fn delete_objects_replication_config_failure_does_not_call_storage_delete(
});
let err = usecase
.execute_delete_objects(req)
.execute_delete_objects_from_s3s(req)
.await
.expect_err("replication config failure must reject DeleteObjects");
+1 -1
View File
@@ -35,7 +35,7 @@ pub(crate) mod object_traffic_health;
pub mod object_usecase;
pub(crate) mod runtime_sources;
mod select_object;
mod select_request;
pub(crate) mod select_request;
pub(crate) mod storage_api;
pub(crate) mod table_list_isolation;
pub(crate) mod trailer_adapter;
+5 -3
View File
@@ -59,8 +59,7 @@ use super::storage_api::multipart_usecase::options::{
};
use super::storage_api::multipart_usecase::request_context::spawn_traced_join;
use super::storage_api::multipart_usecase::s3_api::multipart::{
ListMultipartUploadsParams, build_list_parts_output, parse_list_multipart_uploads_params, parse_list_parts_params,
parse_upload_part_number,
ListMultipartUploadsParams, parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
};
use super::storage_api::multipart_usecase::set_disk::is_valid_storage_class;
use super::storage_api::multipart_usecase::sse::{
@@ -103,7 +102,6 @@ use bytes::Bytes;
use futures::StreamExt;
use http::{HeaderMap, HeaderValue, Uri};
use metrics::counter;
use rustfs_io_metrics::record_s3_op;
use rustfs_s3_ops::S3Operation;
use rustfs_targets::EventName;
use rustfs_utils::CompressionAlgorithm;
@@ -127,11 +125,15 @@ use urlencoding::encode;
use uuid::Uuid;
mod abort;
pub(crate) use abort::{AbortMultipartUploadTarget, reject_abort_capabilities};
mod complete;
mod create;
mod create_values;
pub(crate) use create_values::{CreateMultipartUploadFields, CreateMultipartUploadResult};
mod list;
#[cfg(feature = "gateway-stack")]
pub(crate) use list::ListPartsInfo;
pub(crate) use list::{ListPartsFields, ListPartsTarget};
#[cfg(test)]
mod test_support;
mod upload_part;
+166 -63
View File
@@ -15,76 +15,179 @@
//! AbortMultipartUpload.
use super::*;
use crate::app::storage_api::s3::dto::{AbortMultipartUploadInput, AbortMultipartUploadOutput};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use crate::app::RequestEnvelope;
#[cfg(any(feature = "gateway-stack", test))]
use crate::app::storage_api::multipart_usecase::contract::multipart::{MultipartAbortAuthorization, MultipartOwner};
use http::StatusCode;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, RecordedUpload, UploadIdClaim, resolve_upload, resolve_upload_async};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
#[cfg(any(feature = "gateway-stack", test))]
use std::sync::OnceLock;
impl DefaultMultipartUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_abort_multipart_upload(
&self,
req: S3Request<AbortMultipartUploadInput>,
) -> S3Result<S3Response<AbortMultipartUploadOutput>> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedPresignedRequest>().is_some(),
)?;
record_s3_op(S3Operation::AbortMultipartUpload);
let mut opts = ObjectOptions::default();
apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
let AbortMultipartUploadInput {
bucket, key, upload_id, ..
} = req.input;
pub(crate) enum AbortMultipartUploadTarget {
Legacy {
bucket: String,
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
upload_id: UploadIdClaim,
},
}
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &key).await?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
// Per the AWS S3 API reference, a malformed upload ID is reported as
// NoSuchUpload (404), the same as an upload that was aborted or
// completed. See multipart_upload_api_error.
match store
.abort_multipart_upload(bucket.as_str(), key.as_str(), upload_id.as_str(), &opts)
.await
{
Ok(_) => {
rustfs_scanner::record_dirty_usage_object_from_producer(
&bucket,
&key,
rustfs_scanner::SegmentInvalidationProducerIdentity::AbortMultipartUpload,
);
Ok(S3Response::new(AbortMultipartUploadOutput { ..Default::default() }))
}
Err(err) => Err(multipart_upload_api_error(err)),
impl AbortMultipartUploadTarget {
fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
}
#[cfg(test)]
mod tests {
use super::super::test_support::*;
use super::*;
pub(crate) fn reject_abort_capabilities(envelope: &RequestEnvelope) -> S3Result<()> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
envelope.headers(),
envelope.uri().query(),
envelope.verified_sigv4().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
envelope.headers(),
envelope.uri().query(),
envelope.verified_presigned().is_some(),
)?;
Ok(())
}
#[tokio::test]
async fn execute_abort_multipart_upload_returns_internal_error_when_store_uninitialized() {
let input = AbortMultipartUploadInput::builder()
.bucket("bucket".to_string())
.key("object".to_string())
.upload_id("upload-id".to_string())
.build()
.unwrap();
let req = build_request(input, Method::DELETE);
let before = s3_op_total(S3Operation::AbortMultipartUpload);
impl DefaultMultipartUsecase {
#[instrument(level = "debug", skip(self, envelope, target))]
pub(crate) async fn execute_abort_multipart_upload(
&self,
envelope: RequestEnvelope,
target: S3Result<AbortMultipartUploadTarget>,
) -> S3Result<()> {
RequestEnvelope::audit_only_operation(
envelope,
S3Operation::AbortMultipartUpload,
StatusCode::OK,
|envelope| async move {
let target = target?;
let (bucket, key) = target.resource();
let mut opts = ObjectOptions::default();
apply_bucket_generation_guard(&envelope, bucket, &mut opts)?;
#[cfg(any(feature = "gateway-stack", test))]
if matches!(&target, AbortMultipartUploadTarget::Gateway { .. }) && opts.expected_bucket_incarnation_id.is_none()
{
return Err(S3Error::with_message(S3ErrorCode::InternalError, "bucket generation guard is missing"));
}
let err = make_usecase().execute_abort_multipart_upload(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert_eq!(s3_op_total(S3Operation::AbortMultipartUpload), before + 1);
reject_veeam_sosapi_mutation(|| self.object_store(), bucket, key).await?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
match &target {
AbortMultipartUploadTarget::Legacy { upload_id, .. } => store
.abort_multipart_upload(bucket, key, upload_id, &opts)
.await
.map_err(multipart_upload_api_error)?,
#[cfg(any(feature = "gateway-stack", test))]
AbortMultipartUploadTarget::Gateway { bucket, key, upload_id } => {
abort_gateway(&store, bucket, key, upload_id, opts).await?
}
}
rustfs_scanner::record_dirty_usage_object_from_producer(
bucket,
key,
rustfs_scanner::SegmentInvalidationProducerIdentity::AbortMultipartUpload,
);
Ok(())
},
)
.await
}
}
#[cfg(any(feature = "gateway-stack", test))]
struct AbortOwner(MultipartOwner);
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for AbortOwner {
fn bucket(&self) -> &str {
&self.0.bucket
}
fn key(&self) -> &str {
&self.0.object
}
}
#[cfg(any(feature = "gateway-stack", test))]
struct AbortAuthorization {
bucket: BucketName,
key: ObjectKey,
upload_id: UploadIdClaim,
observed: OnceLock<MultipartOwner>,
}
#[cfg(any(feature = "gateway-stack", test))]
impl MultipartAbortAuthorization for AbortAuthorization {
type Error = StorageError;
fn authorize(&self, owner: Option<&MultipartOwner>) -> Result<(), Self::Error> {
let (_, owner) =
resolve_upload(&self.upload_id, &self.bucket, &self.key, |_| owner.cloned().map(AbortOwner)).map_err(|_| {
StorageError::InvalidUploadID(self.bucket.as_str().to_owned(), self.key.as_str().to_owned(), String::new())
})?;
self.observed.get_or_init(|| owner.0);
Ok(())
}
}
#[cfg(any(feature = "gateway-stack", test))]
async fn abort_gateway(
store: &ECStore,
bucket: &BucketName,
key: &ObjectKey,
upload_id: &UploadIdClaim,
mut opts: ObjectOptions,
) -> S3Result<()> {
let authorization = Arc::new(AbortAuthorization {
bucket: bucket.clone(),
key: key.clone(),
upload_id: upload_id.clone(),
observed: OnceLock::new(),
});
opts.multipart_abort_authorization = Some(authorization.clone());
// The outer exchange checks the claim's shape. Only the inner exchange, on the
// selected metadata while holding the upload lock, authorizes deletion.
resolve_upload_async(upload_id, bucket, key, |candidate| async move {
store
.abort_multipart_upload(bucket.as_str(), key.as_str(), candidate, &opts)
.await
.map_err(multipart_upload_api_error)?;
let owner = authorization.observed.get().cloned().ok_or_else(|| {
S3Error::with_message(S3ErrorCode::InternalError, "Abort authorization did not observe the upload owner")
})?;
Ok::<_, S3Error>(Some(AbortOwner(owner)))
})
.await?
.map_err(|_| {
// An empty id has the legacy missing-upload response. Other impossible shapes
// have the malformed-upload response. Neither reaches the storage lookup.
let error = if upload_id.is_explicitly_empty() {
StorageError::InvalidUploadID(bucket.as_str().to_owned(), key.as_str().to_owned(), String::new())
} else {
StorageError::MalformedUploadID(String::new())
};
multipart_upload_api_error(error)
})?;
Ok(())
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,205 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::super::test_support::*;
use super::*;
use crate::app::storage_api::multipart_usecase::access::{AuthorizationRequest, load_bucket_generation_from_store};
use crate::app::storage_api::multipart_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::multipart_usecase::request_context::RequestContext;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use crate::runtime_sources::ServerContextSlot;
use rustfs_scanner::{SegmentInvalidationProducerIdentity, set_scanner_dirty_usage_mutation_observer};
use std::sync::atomic::{AtomicUsize, Ordering};
#[tokio::test]
async fn execute_abort_multipart_upload_returns_internal_error_when_store_uninitialized() {
let request = build_request((), Method::DELETE);
let envelope = RequestEnvelope::from_s3s(&request);
let before = s3_op_total(S3Operation::AbortMultipartUpload);
let target = AbortMultipartUploadTarget::Legacy {
bucket: "bucket".to_owned(),
key: "object".to_owned(),
upload_id: "upload-id".to_owned(),
};
let err = make_usecase()
.execute_abort_multipart_upload(envelope, Ok(target))
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert_eq!(err.message(), Some("Not init"));
assert_eq!(s3_op_total(S3Operation::AbortMultipartUpload), before);
}
#[test]
#[serial_test::serial]
fn native_abort_requires_current_generation_and_audits_each_result() {
crate::app::gating_test_env::run_large_stack_test("abort-multipart-generation", || async {
let context = crate::app::gating_test_env::shared_gating_ambient().await;
let store = context.object_store();
let usecase = DefaultMultipartUsecase::with_context(Some(context.clone()));
let slot = ServerContextSlot::new();
assert!(slot.install(context));
let bucket = format!("abort-multipart-{}", Uuid::new_v4().simple());
let other = format!("abort-multipart-{}", Uuid::new_v4().simple());
let key = "object";
let opts = ObjectOptions::default();
let request_context = RequestContext::fallback();
let request = || {
let mut req = build_request((), Method::DELETE);
req.uri = format!("/{bucket}/{key}").parse().expect("fixture URI");
req.extensions.insert(slot.clone());
req.extensions.insert(request_context.clone());
req
};
for name in [&bucket, &other] {
store
.make_bucket(name, &MakeBucketOptions::default())
.await
.expect("create bucket");
}
let original = store
.bucket_incarnation_id(&bucket)
.await
.expect("previous bucket incarnation");
let mut stale = request();
let captured = load_bucket_generation_from_store(&store, &stale, &bucket)
.await
.expect("capture prior bucket");
stale.install_bucket_generation(captured);
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
assert_ne!(
store
.bucket_incarnation_id(&bucket)
.await
.expect("current bucket incarnation"),
original
);
let upload = store
.new_multipart_upload(&bucket, key, &opts)
.await
.expect("create current upload");
let mut reader = PutObjReader::from_vec(b"current part".to_vec());
store
.put_object_part(&bucket, key, &upload.upload_id, 1, &mut reader, &opts)
.await
.expect("stage current part");
let target = || AbortMultipartUploadTarget::Gateway {
bucket: BucketName::new(bucket.clone()).expect("bucket name"),
key: ObjectKey::new(key).expect("object key"),
upload_id: UploadIdClaim::from_wire(upload.upload_id.as_str()),
};
let dirty = Arc::new(AtomicUsize::new(0));
let observer = {
let dirty = dirty.clone();
let bucket = bucket.clone();
Arc::new(move |observed_bucket: &str, observed_key: &str, producer| {
if observed_bucket == bucket
&& observed_key == key
&& producer == SegmentInvalidationProducerIdentity::AbortMultipartUpload
{
dirty.fetch_add(1, Ordering::Relaxed);
}
}) as rustfs_scanner::ScannerDirtyUsageMutationObserver
};
struct RestoreObserver(Option<rustfs_scanner::ScannerDirtyUsageMutationObserver>);
impl Drop for RestoreObserver {
fn drop(&mut self) {
set_scanner_dirty_usage_mutation_observer(self.0.take());
}
}
let _observer = RestoreObserver(set_scanner_dirty_usage_mutation_observer(Some(observer)));
let probe = MutationCompletionProbe::install(&bucket);
let missing = request();
let mut wrong = request();
let captured = load_bucket_generation_from_store(&store, &wrong, &other)
.await
.expect("capture unrelated bucket");
wrong.install_bucket_generation(captured);
for (req, code) in [
(stale, S3ErrorCode::NoSuchBucket),
(missing, S3ErrorCode::InternalError),
(wrong, S3ErrorCode::InternalError),
] {
let err = usecase
.execute_abort_multipart_upload(RequestEnvelope::from_s3s(&req), Ok(target()))
.await
.expect_err("Abort requires current generation");
assert_eq!(err.code(), &code);
let parts = store
.list_object_parts(&bucket, key, &upload.upload_id, None, 10, &opts)
.await
.expect("current upload persists");
assert_eq!(parts.parts.len(), 1, "current staged part persists");
}
assert_eq!(dirty.load(Ordering::Relaxed), 0);
assert_eq!(probe.audits.lock().unwrap().len(), 3);
let mut current = request();
let captured = load_bucket_generation_from_store(&store, &current, &bucket)
.await
.expect("capture current bucket");
current.install_bucket_generation(captured);
usecase
.execute_abort_multipart_upload(RequestEnvelope::from_s3s(&current), Ok(target()))
.await
.expect("current generation allows Abort");
assert!(matches!(
store.get_multipart_info(&bucket, key, &upload.upload_id, &opts).await,
Err(StorageError::InvalidUploadID(..))
));
assert_eq!(dirty.load(Ordering::Relaxed), 1);
{
let audits = probe.audits.lock().unwrap();
assert_eq!(audits.len(), 4);
assert_eq!(audits.last().expect("successful audit").api.status_code, Some(200));
}
let legacy = store
.new_multipart_upload(&bucket, key, &opts)
.await
.expect("create legacy-path upload");
usecase
.execute_abort_multipart_upload(
RequestEnvelope::from_s3s(&current),
Ok(AbortMultipartUploadTarget::Legacy {
bucket: bucket.clone(),
key: key.to_owned(),
upload_id: legacy.upload_id,
}),
)
.await
.expect("legacy target with the same authorized generation");
assert_eq!(dirty.load(Ordering::Relaxed), 2);
let audits = probe.audits.lock().unwrap();
assert_eq!(audits.len(), 5, "one audit per owned Abort");
assert_eq!(audits.last().expect("legacy success audit").api.status_code, Some(200));
for audit in audits.iter() {
assert_eq!(audit.request_id.as_deref(), Some(request_context.request_id.as_str()));
assert_eq!(audit.api.name.as_deref(), Some(S3Operation::AbortMultipartUpload.as_str()));
}
assert!(probe.events.lock().unwrap().is_empty());
});
}
+139 -56
View File
@@ -18,9 +18,82 @@ use super::*;
use crate::app::RequestEnvelope;
use crate::app::dto_values;
use crate::app::storage_api::multipart_usecase::contract::multipart::ListMultipartsInfo;
use crate::app::storage_api::s3::dto::{ListPartsInput, ListPartsOutput};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
pub(crate) use crate::app::storage_api::multipart_usecase::contract::multipart::ListPartsInfo;
#[cfg(any(feature = "gateway-stack", test))]
use crate::app::storage_api::multipart_usecase::contract::multipart::ListPartsOwner;
use rustfs_gateway_types::dto as gateway_dto;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, RecordedUpload, UploadIdClaim, resolve_upload_async};
use rustfs_s3_types::S3Result;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_s3_types::{S3Error, S3ErrorCode};
pub(crate) enum ListPartsTarget {
Legacy {
bucket: String,
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
upload_id: UploadIdClaim,
},
}
impl ListPartsTarget {
fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
}
pub(crate) struct ListPartsFields {
pub(crate) target: ListPartsTarget,
pub(crate) part_number_marker: Option<String>,
pub(crate) max_parts: Option<i32>,
}
#[cfg(any(feature = "gateway-stack", test))]
struct RecordedListParts {
owner: ListPartsOwner,
page: ListPartsInfo,
}
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for RecordedListParts {
fn bucket(&self) -> &str {
&self.owner.bucket
}
fn key(&self) -> &str {
&self.owner.object
}
}
#[cfg(any(feature = "gateway-stack", test))]
async fn resolve_list_parts<'a, F>(
upload_id: &'a UploadIdClaim,
bucket: &BucketName,
key: &ObjectKey,
lookup: impl FnOnce(&'a str) -> F,
) -> S3Result<ListPartsInfo>
where
F: Future<Output = Result<ListPartsInfo, StorageError>>,
{
let (_, record) = resolve_upload_async(upload_id, bucket, key, |id| async move {
let mut page = lookup(id).await?;
Ok::<_, StorageError>(page.recorded_owner.take().map(|owner| RecordedListParts { owner, page }))
})
.await
.map_err(|error| S3Error::from(multipart_upload_api_error(error)))?
.map_err(|refusal| S3Error::with_message(S3ErrorCode::NoSuchUpload, refusal.reason()))?;
Ok(record.page)
}
impl DefaultMultipartUsecase {
/// One page of ListMultipartUploads: the uploads of the bucket's current incarnation.
@@ -101,41 +174,58 @@ impl DefaultMultipartUsecase {
list_multipart_uploads_output(bucket, prefix, result, input.encoding_type)
}
pub async fn execute_list_parts(&self, req: S3Request<ListPartsInput>) -> S3Result<S3Response<ListPartsOutput>> {
pub(crate) async fn execute_list_parts(&self, envelope: RequestEnvelope, fields: ListPartsFields) -> S3Result<ListPartsInfo> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
envelope.headers(),
envelope.uri().query(),
envelope.verified_sigv4().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedPresignedRequest>().is_some(),
envelope.headers(),
envelope.uri().query(),
envelope.verified_presigned().is_some(),
)?;
let (bucket, key) = fields.target.resource();
let mut opts = ObjectOptions::default();
apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
let ListPartsInput {
bucket,
key,
upload_id,
part_number_marker,
max_parts,
..
} = req.input;
apply_bucket_generation_guard(&envelope, bucket, &mut opts)?;
reject_veeam_sosapi_mutation(|| self.object_store(), bucket, key).await?;
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &key).await?;
let params = parse_list_parts_params(part_number_marker, max_parts)?;
let marker = fields
.part_number_marker
.as_deref()
.map(|marker| {
marker
.parse::<i32>()
.map_err(|_| rustfs_s3_types::s3_error!(InvalidArgument, "invalid query: part-number-marker: {marker}"))
})
.transpose()?;
let params = parse_list_parts_params(marker, fields.max_parts)?;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
let res = store
.list_object_parts(&bucket, &key, &upload_id, params.part_number_marker, params.max_parts, &opts)
.await
.map_err(multipart_upload_api_error)?;
Ok(S3Response::new(build_list_parts_output(res)))
match &fields.target {
ListPartsTarget::Legacy { upload_id, .. } => store
.list_object_parts(bucket, key, upload_id, params.part_number_marker, params.max_parts, &opts)
.await
.map_err(|error| multipart_upload_api_error(error).into()),
#[cfg(any(feature = "gateway-stack", test))]
ListPartsTarget::Gateway {
bucket: bucket_name,
key: object_key,
upload_id,
} => {
let store = &store;
let opts = &opts;
resolve_list_parts(upload_id, bucket_name, object_key, |id| async move {
store
.list_object_parts(bucket, key, id, params.part_number_marker, params.max_parts, opts)
.await
})
.await
}
}
}
}
@@ -192,6 +282,9 @@ fn list_multipart_uploads_output(
})
}
#[cfg(test)]
mod parts_tests;
#[cfg(test)]
mod tests {
use super::super::test_support::*;
@@ -301,48 +394,38 @@ mod tests {
assert!(output.common_prefixes.is_empty());
}
async fn list_parts_refusal(marker: Option<&str>, max_parts: Option<i32>) -> S3Error {
let fields = ListPartsFields {
target: ListPartsTarget::Legacy {
bucket: "bucket".to_owned(),
key: "object".to_owned(),
upload_id: "upload-id".to_owned(),
},
part_number_marker: marker.map(str::to_owned),
max_parts,
};
make_usecase()
.execute_list_parts(RequestEnvelope::from_s3s(&build_request((), Method::GET)), fields)
.await
.expect_err("refused")
}
#[tokio::test]
async fn execute_list_parts_returns_internal_error_when_store_uninitialized() {
let input = ListPartsInput::builder()
.bucket("bucket".to_string())
.key("object".to_string())
.upload_id("upload-id".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let err = make_usecase().execute_list_parts(req).await.unwrap_err();
let err = list_parts_refusal(None, None).await;
assert_eq!(err.code(), &S3ErrorCode::InternalError);
}
#[tokio::test]
async fn execute_list_parts_rejects_negative_part_number_marker_before_store_lookup() {
let input = ListPartsInput::builder()
.bucket("bucket".to_string())
.key("object".to_string())
.upload_id("upload-id".to_string())
.part_number_marker(Some(-1))
.build()
.unwrap();
let req = build_request(input, Method::GET);
let err = make_usecase().execute_list_parts(req).await.unwrap_err();
let err = list_parts_refusal(Some("-1"), Some(0)).await;
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(err.message(), Some("part-number-marker must be non-negative"));
}
#[tokio::test]
async fn execute_list_parts_rejects_invalid_max_parts_before_store_lookup() {
let input = ListPartsInput::builder()
.bucket("bucket".to_string())
.key("object".to_string())
.upload_id("upload-id".to_string())
.max_parts(Some(1001))
.build()
.unwrap();
let req = build_request(input, Method::GET);
let err = make_usecase().execute_list_parts(req).await.unwrap_err();
let err = list_parts_refusal(None, Some(1001)).await;
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(err.message(), Some("max-parts must be between 1 and 1000"));
}
@@ -0,0 +1,310 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! ListParts execution against real multipart metadata and the native claim exchange.
use super::super::test_support::{Method, build_request};
use super::*;
use crate::app::gating_test_env::{run_large_stack_test, shared_gating_ambient, shared_gating_ecstore_and_disk_paths};
use crate::app::storage_api::multipart_usecase::access::{AuthorizationRequest, load_bucket_generation_from_store};
use crate::app::storage_api::s3::S3Error as LegacyS3Error;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use crate::runtime_sources::ServerContextSlot;
use sha2::{Digest, Sha256};
use std::sync::atomic::{AtomicUsize, Ordering::Relaxed};
use tracing::instrument::WithSubscriber as _;
use tracing::{Subscriber, span};
use tracing_subscriber::{Layer, layer::Context, layer::SubscriberExt as _};
#[derive(Clone, Default)]
struct UploadLookups(Arc<AtomicUsize>);
impl<S: Subscriber> Layer<S> for UploadLookups {
fn on_new_span(&self, attrs: &span::Attributes<'_>, _: &span::Id, _: Context<'_, S>) {
let meta = attrs.metadata();
if meta
.file()
.is_some_and(|file| file.ends_with("crates/ecstore/src/store/mod.rs"))
&& matches!(meta.name(), "list_object_parts" | "get_multipart_info")
{
self.0.fetch_add(1, Relaxed);
}
}
}
fn gateway_fields(bucket: &str, key: &str, id: &str, marker: Option<&str>, max_parts: Option<i32>) -> ListPartsFields {
ListPartsFields {
target: ListPartsTarget::Gateway {
bucket: BucketName::new(bucket).expect("fixture bucket"),
key: ObjectKey::new(key).expect("fixture key"),
upload_id: UploadIdClaim::from_wire(id),
},
part_number_marker: marker.map(str::to_owned),
max_parts,
}
}
async fn execute_counted(
usecase: &DefaultMultipartUsecase,
envelope: RequestEnvelope,
fields: ListPartsFields,
expected: usize,
) -> S3Result<ListPartsInfo> {
let lookups = UploadLookups::default();
let result = usecase
.execute_list_parts(envelope, fields)
.with_subscriber(tracing_subscriber::registry().with(lookups.clone()))
.await;
assert_eq!(lookups.0.load(Relaxed), expected, "storage upload lookups");
result
}
#[test]
#[serial_test::serial]
fn list_parts_resolves_real_owners_and_retains_storage_errors() {
run_large_stack_test("list-parts-ownership", || async {
let (paths, store) = shared_gating_ecstore_and_disk_paths().await;
let usecase = DefaultMultipartUsecase::with_context(Some(shared_gating_ambient().await));
let bucket = format!("list-parts-{}", Uuid::new_v4().simple());
let foreign_bucket = format!("list-parts-{}", Uuid::new_v4().simple());
let key = "fixture/one";
let foreign_key = "fixture/two";
let opts = ObjectOptions::default();
let envelope = || RequestEnvelope::from_s3s(&build_request((), Method::GET));
for name in [&bucket, &foreign_bucket] {
store
.make_bucket(name, &MakeBucketOptions::default())
.await
.expect("create a fixture bucket");
}
let upload = store
.new_multipart_upload(&bucket, key, &opts)
.await
.expect("create an active session");
let id = upload.upload_id.as_str();
for (claim, marker, max, lookups, code) in [
(id, None, None, 1, None),
("", None, None, 0, Some("NoSuchUpload")),
("/impossible", None, None, 0, Some("NoSuchUpload")),
("malformed", None, None, 1, Some("NoSuchUpload")),
("/impossible", Some("-1"), Some(0), 0, Some("InvalidArgument")),
("/impossible", None, Some(0), 0, Some("InvalidArgument")),
] {
let result = execute_counted(&usecase, envelope(), gateway_fields(&bucket, key, claim, marker, max), lookups).await;
assert_eq!(
result.as_ref().err().map(|err| err.code().as_str()),
code,
"marker {marker:?}, max {max:?}"
);
if let Ok(page) = result {
assert!(page.parts.is_empty());
assert!(page.upload_id == id);
}
}
let typed_bucket = BucketName::new(bucket.clone()).unwrap();
let typed_key = ObjectKey::new(key).unwrap();
let claim = UploadIdClaim::from_wire(id);
for (owner_bucket, owner_key) in [(foreign_bucket.as_str(), key), (bucket.as_str(), foreign_key)] {
let other = store
.new_multipart_upload(owner_bucket, owner_key, &opts)
.await
.expect("create an independent active session");
let mut actual_page = store
.list_object_parts(owner_bucket, owner_key, &other.upload_id, None, 1, &opts)
.await
.expect("the independent session returns a real page");
// Only the durable owner observation is relevant, even if presentation agrees.
actual_page.bucket.clone_from(&bucket);
actual_page.object = key.to_owned();
let refusal = resolve_list_parts(&claim, &typed_bucket, &typed_key, |supplied| {
assert!(supplied == id, "the callback receives the original upload ID");
std::future::ready(Ok(actual_page))
})
.await
.expect_err("a foreign recorded owner cannot be returned");
assert_eq!(refusal.code(), &S3ErrorCode::NoSuchUpload);
}
let mut unproved = store
.list_object_parts(&bucket, key, id, None, 1, &opts)
.await
.expect("live session");
unproved.recorded_owner = None;
let refusal = resolve_list_parts(&claim, &typed_bucket, &typed_key, |_| std::future::ready(Ok(unproved)))
.await
.expect_err("a page without owner proof must be refused");
assert_eq!(refusal.code(), &S3ErrorCode::NoSuchUpload);
let staging_name = hex_simd::encode_to_string(Sha256::digest(format!("{bucket}/{key}")), hex_simd::AsciiCase::Lower);
for disk in &paths[..3] {
let staging = disk.join(".rustfs.sys/multipart").join(&staging_name);
let uploads: Vec<_> = std::fs::read_dir(&staging).expect("real upload storage namespace").collect();
assert_eq!(uploads.len(), 1, "the test owns this upload namespace");
let path = uploads.into_iter().next().unwrap().unwrap().path().join("xl.meta");
assert!(path.is_file(), "the upload has persisted metadata");
std::fs::write(path, b"unreadable test metadata").expect("damage the fixture metadata only");
}
let storage_error = store
.list_object_parts(&bucket, key, id, None, 1000, &opts)
.await
.expect_err("loss of three upload metadata replicas fails the read");
assert!(!matches!(
storage_error,
StorageError::InvalidUploadID(..) | StorageError::MalformedUploadID(..)
));
let expected = multipart_upload_api_error(storage_error);
for target in [
gateway_fields(&bucket, key, id, None, None),
ListPartsFields {
target: ListPartsTarget::Legacy {
bucket: bucket.clone(),
key: key.to_owned(),
upload_id: id.to_owned(),
},
part_number_marker: None,
max_parts: None,
},
] {
let refusal = execute_counted(&usecase, envelope(), target, 1)
.await
.expect_err("surface the outage");
let legacy: LegacyS3Error = refusal.into();
assert_eq!(legacy.code(), expected.code());
assert_eq!(legacy.status_code(), expected.status_code());
assert_eq!(legacy.message(), expected.message());
assert!(
legacy
.source()
.and_then(|cause| cause.downcast_ref::<StorageError>())
.is_some()
);
}
});
}
#[test]
#[serial_test::serial]
fn list_parts_applies_the_authorized_generation_before_lookup() {
run_large_stack_test("list-parts-generation", || async {
let context = shared_gating_ambient().await;
let store = context.object_store();
let usecase = DefaultMultipartUsecase::with_context(Some(context.clone()));
let slot = ServerContextSlot::new();
assert!(slot.install(context));
let bucket = format!("list-parts-{}", Uuid::new_v4().simple());
let other = format!("list-parts-{}", Uuid::new_v4().simple());
let key = "fixture";
let opts = ObjectOptions::default();
for name in [&bucket, &other] {
store
.make_bucket(name, &MakeBucketOptions::default())
.await
.expect("create a fixture bucket");
}
let req = || {
let mut req = build_request((), Method::GET);
req.extensions.insert(slot.clone());
req
};
let capability = |query: &str| {
let mut request = req();
request.uri = format!("/fixture?{query}").parse().expect("fixture URI");
RequestEnvelope::from_s3s(&request)
};
for (query, first) in [
(
"x-rustfs-max-total-object-size=1&x-rustfs-max-content-length=1",
"x-rustfs-max-total-object-size",
),
("x-rustfs-max-content-length=1", "x-rustfs-max-content-length"),
] {
let err = execute_counted(
&usecase,
capability(query),
gateway_fields(&bucket, key, "/impossible", Some("-1"), None),
0,
)
.await
.expect_err("capability validation precedes generation, pagination and claim resolution");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
assert!(err.message().is_some_and(|message| message.starts_with(first)));
}
let old = load_bucket_generation_from_store(&store, &req(), &bucket)
.await
.expect("old generation");
let other_guard = load_bucket_generation_from_store(&store, &req(), &other)
.await
.expect("other generation");
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("remove the original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate the bucket");
let upload = store
.new_multipart_upload(&bucket, key, &opts)
.await
.expect("session from the recreated bucket");
let current = load_bucket_generation_from_store(&store, &req(), &bucket)
.await
.expect("current generation");
let mut veeam_request = req();
veeam_request.install_bucket_generation(current.clone());
temp_env::async_with_vars([(rustfs_config::ENV_VEEAM_SOSAPI_ENABLE, Some("true"))], async {
let err = execute_counted(
&usecase,
RequestEnvelope::from_s3s(&veeam_request),
gateway_fields(
&bucket,
".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml",
"/impossible",
Some("-1"),
None,
),
0,
)
.await
.expect_err("reserved objects are checked before pagination and upload claims");
assert_eq!(err.code(), &S3ErrorCode::MethodNotAllowed);
})
.await;
for (guard, marker, lookups, code) in [
(Some(old), None, 1, Some("NoSuchBucket")),
(Some(other_guard), Some("-1"), 0, Some("InternalError")),
(None, Some("-1"), 0, Some("InternalError")),
(Some(current), None, 1, None),
] {
let mut request = req();
if let Some(guard) = guard {
request.install_bucket_generation(guard);
}
let result = execute_counted(
&usecase,
RequestEnvelope::from_s3s(&request),
gateway_fields(&bucket, key, &upload.upload_id, marker, None),
lookups,
)
.await;
assert_eq!(result.as_ref().err().map(|err| err.code().as_str()), code);
if let Ok(page) = result {
assert!(page.upload_id == upload.upload_id);
}
}
});
}
+299
View File
@@ -0,0 +1,299 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Read the stored object attributes requested by either S3 edge.
use super::head::ResponseEtag;
use super::*;
use crate::app::{Answered, RequestEnvelope};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
const ETAG: &str = "ETag";
const CHECKSUM: &str = "Checksum";
const OBJECT_PARTS: &str = "ObjectParts";
const STORAGE_CLASS: &str = "StorageClass";
const OBJECT_SIZE: &str = "ObjectSize";
/// Decoded inputs preserve the version spelling and the raw attribute choices until use.
/// Customer key material must never be rendered as Debug output.
pub(crate) struct GetObjectAttributesFields {
pub(crate) bucket: String,
pub(crate) key: String,
pub(crate) version_id: Option<String>,
pub(crate) object_attributes: Vec<String>,
pub(crate) max_parts: Option<i32>,
pub(crate) part_number_marker: Option<String>,
pub(crate) sse_customer_key: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
}
/// Stored values are encoded by the response edge after the operation completes.
pub(crate) struct GetObjectAttributesOutputFacts {
pub(crate) e_tag: Option<ResponseEtag>,
pub(crate) last_modified: Option<OffsetDateTime>,
pub(crate) version_id: Option<String>,
pub(crate) storage_class: Option<String>,
pub(crate) object_size: Option<i64>,
pub(crate) checksum: Option<ResponseChecksums>,
pub(crate) object_parts: Option<AttributesPartsFacts>,
}
pub(crate) struct AttributesPartsFacts {
pub(crate) is_truncated: bool,
pub(crate) max_parts: i32,
pub(crate) next_part_number_marker: Option<i32>,
pub(crate) part_number_marker: Option<i32>,
pub(crate) parts: Vec<AttributesPartFacts>,
pub(crate) total_parts_count: i32,
}
pub(crate) struct AttributesPartFacts {
pub(crate) part_number: Option<i32>,
pub(crate) size: i64,
pub(crate) checksums: ResponseChecksums,
}
impl DefaultObjectUsecase {
pub(crate) async fn execute_get_object_attributes(
&self,
envelope: RequestEnvelope,
input: GetObjectAttributesFields,
) -> S3Result<Answered<GetObjectAttributesOutputFacts>> {
if let Some(context) = &self.context {
let _ = context.object_store();
}
let mut helper = OperationHelper::from_view(
&envelope.audit_view(),
EventName::ObjectAccessedAttributes,
S3Operation::GetObjectAttributes,
)
.suppress_event();
let GetObjectAttributesFields {
bucket,
key,
version_id,
object_attributes,
max_parts,
part_number_marker,
sse_customer_key,
sse_customer_key_md5,
} = input;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init"));
};
let headers = envelope.headers();
let mut opts = get_opts(&bucket, &key, version_id, None, headers)
.await
.map_err(ApiError::from)?;
let requested = |name| object_attributes_requested(&object_attributes, name);
opts.include_part_checksums = requested(OBJECT_PARTS);
let info = match store.get_object_info(&bucket, &key, &opts).await {
Ok(info) => info,
Err(err) => {
if is_err_object_not_found(&err) || is_err_version_not_found(&err) {
if is_dir_object(&key) {
let has_children = match probe_prefix_has_children(store, &bucket, &key, false).await {
Ok(has_children) => has_children,
Err(e) => {
error!(
"Failed to probe children for object attributes (bucket: {}, key: {}): {}",
bucket, key, e
);
false
}
};
let msg = head_prefix_not_found_message(&bucket, &key, has_children);
return Err(S3Error::with_message(S3ErrorCode::NoSuchKey, msg));
}
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
}
return Err(ApiError::from(err).into());
}
};
if info.delete_marker {
if opts.version_id.is_none() {
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
}
return Err(S3Error::new(S3ErrorCode::MethodNotAllowed));
}
validate_ssec_for_read(&info.user_defined, sse_customer_key.as_ref(), sse_customer_key_md5.as_ref())?;
let storage_class = response_storage_class_for_object_attributes(&info, &info.user_defined, requested(STORAGE_CLASS));
let e_tag = if requested(ETAG) {
info.etag.as_deref().map(ResponseEtag::from_stored)
} else {
None
};
let object_size = if requested(OBJECT_SIZE) {
Some(info.get_actual_size().map_err(ApiError::from)?)
} else {
None
};
let checksum = if requested(CHECKSUM) {
let (checksums, is_multipart) = info.decrypt_checksums(0, headers).map_err(ApiError::from)?;
Some(classify_response_checksums(checksums, is_multipart))
} else {
None
};
let object_parts = if requested(OBJECT_PARTS) && info.is_multipart() {
Some(read_parts(&info, headers, part_number_marker.as_deref(), max_parts)?)
} else {
None
};
let version_id = s3_response_version_id(info.version_id);
let output = GetObjectAttributesOutputFacts {
e_tag,
last_modified: info.mod_time,
version_id: version_id.clone(),
storage_class,
object_size,
checksum,
object_parts,
};
helper = helper.object(info).version_id(version_id.unwrap_or_default());
let result = Ok(Answered::new(output));
let _ = helper.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
result
}
}
fn read_parts(
info: &ObjectInfo,
headers: &HeaderMap,
part_number_marker: Option<&str>,
max_parts: Option<i32>,
) -> S3Result<AttributesPartsFacts> {
// Legacy headers have already decoded this as an i32. Native marker text is
// only parsed when a stored multipart object actually needs a page.
let part_number_marker = part_number_marker
.map(str::parse::<i32>)
.transpose()
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "Invalid part-number-marker"))?;
let params = parse_list_parts_params(part_number_marker, max_parts)?;
let mut parts = Vec::new();
let mut marker = params.part_number_marker;
let mut start_at = 0;
if let Some(value) = marker {
if let Some(index) = info.parts.iter().position(|part| part.number == value) {
start_at = index + 1;
} else {
marker = None;
}
}
let max_parts = i32::try_from(params.max_parts)
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "max-parts value is out of range"))?;
let end = (start_at + params.max_parts).min(info.parts.len());
for part in &info.parts[start_at..end] {
let (checksums, is_multipart) = info.decrypt_checksums(part.number, headers).map_err(ApiError::from)?;
let checksums = classify_response_checksums(checksums, is_multipart);
let size = if part.actual_size > 0 {
part.actual_size
} else {
part.size
.try_into()
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "Part size value is out of range"))?
};
parts.push(AttributesPartFacts {
part_number: i32::try_from(part.number).ok(),
size,
checksums,
});
}
Ok(AttributesPartsFacts {
is_truncated: end < info.parts.len(),
max_parts,
next_part_number_marker: parts.last().and_then(|part| part.part_number),
part_number_marker: marker.and_then(|value| i32::try_from(value).ok()),
parts,
total_parts_count: i32::try_from(info.parts.len())
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "Part count is out of range"))?,
})
}
fn object_attributes_requested(object_attributes: &[String], name: &str) -> bool {
object_attributes.iter().any(|value| {
value.split(',').any(|part| {
part.trim_matches(|c: char| c.is_whitespace() || c == '"' || c == '\'')
.eq_ignore_ascii_case(name)
})
})
}
#[cfg(test)]
mod tests {
use super::*;
#[tokio::test]
async fn execute_get_object_attributes_returns_internal_error_when_store_uninitialized() {
let fields = GetObjectAttributesFields {
bucket: "test-bucket".to_owned(),
key: "test-key".to_owned(),
version_id: None,
object_attributes: vec![],
max_parts: None,
part_number_marker: None,
sse_customer_key: None,
sse_customer_key_md5: None,
};
let request = build_request((), http::Method::GET);
let envelope = RequestEnvelope::from_s3s(&request);
let error = DefaultObjectUsecase::without_context()
.execute_get_object_attributes(envelope, fields)
.await
.err()
.expect("uninitialized store");
assert_eq!(error.code(), &S3ErrorCode::InternalError);
}
#[test]
fn object_attributes_requested_with_single_value() {
let selection = vec![ETAG.to_owned()];
assert!(object_attributes_requested(&selection, ETAG));
assert!(!object_attributes_requested(&selection, OBJECT_SIZE));
}
#[test]
fn object_attributes_requested_with_comma_separated_values() {
let selection = vec!["ObjectParts,etag".to_owned(), "StorageClass".to_owned()];
assert!(object_attributes_requested(&selection, OBJECT_PARTS));
assert!(object_attributes_requested(&selection, ETAG));
assert!(!object_attributes_requested(&selection, OBJECT_SIZE));
}
#[test]
fn object_attributes_requested_with_quotes_and_spaces() {
let selection = vec!["'ObjectSize', \"Checksum\" , \"Etag\"".to_owned()];
assert!(object_attributes_requested(&selection, OBJECT_SIZE));
assert!(object_attributes_requested(&selection, CHECKSUM));
assert!(object_attributes_requested(&selection, ETAG));
}
#[test]
fn object_attributes_requested_returns_false_for_missing_name() {
let selection = vec!["Checksum".to_owned()];
assert!(!object_attributes_requested(&selection, OBJECT_SIZE));
}
}
@@ -0,0 +1,117 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Observe GetObjectAttributes completion at the real server's audit and event boundary.
use bytes::Bytes;
use http_body_util::{BodyExt as _, Empty};
use hyper_util::rt::TokioIo;
fn get_object_attributes_completion(stack: Option<&'static str>) {
temp_env::with_vars([("RUSTFS_S3_STACK", stack), ("RUSTFS_AUDIT_ENABLE", Some("true"))], || {
crate::app::gating_test_env::run_large_stack_test("attributes-completion", || async {
use crate::app::storage_api::object_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::test::contract::object::ObjectIO as _;
use crate::embedded::{RustFSServerBuilder, find_available_port};
let server = RustFSServerBuilder::new()
.address(format!("127.0.0.1:{}", find_available_port().unwrap()))
.access_key("attributesprobe")
.secret_key("attributes-fixture-test-secret")
.build()
.await
.unwrap();
crate::server::refresh_audit_module_enabled();
let context = crate::app::runtime_sources::current_app_context().unwrap();
let store = context.object_store();
let bucket = "attributes-completion";
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": ["s3:GetObject", "s3:GetObjectAttributes"],
"Resource": [format!("arn:aws:s3:::{bucket}/object"), format!("arn:aws:s3:::{bucket}/missing")],
}],
});
store
.update_bucket_metadata_config(
bucket,
crate::app::storage_api::object_usecase::bucket::metadata::BUCKET_POLICY_CONFIG,
serde_json::to_vec(&policy).unwrap(),
)
.await
.unwrap();
store
.put_object(
bucket,
"object",
&mut crate::app::storage_api::object_usecase::StoragePutObjReader::from_vec(b"payload".to_vec()),
&Default::default(),
)
.await
.unwrap();
let probe = MutationCompletionProbe::install(bucket);
for (key, expected) in [("object", 200), ("missing", 404), ("other", 403)] {
let socket = tokio::net::TcpStream::connect(server.address()).await.unwrap();
let (mut sender, connection) = hyper::client::conn::http1::handshake(TokioIo::new(socket)).await.unwrap();
let transport = tokio::spawn(connection);
let request = http::Request::get(format!("/{bucket}/{key}?attributes"))
.header(http::header::HOST, "localhost")
.header("x-amz-object-attributes", "ObjectSize")
.body(Empty::<Bytes>::new())
.unwrap();
let response = sender.send_request(request).await.unwrap();
assert_eq!(response.status().as_u16(), expected);
response.into_body().collect().await.unwrap();
drop(sender);
transport.abort();
let _ = transport.await;
}
probe.wait_for(2, 0).await;
let audits = probe
.audits
.lock()
.unwrap()
.iter()
.filter(|entry| entry.api.name.as_deref() == Some(rustfs_s3_ops::S3Operation::GetObjectAttributes.as_str()))
.map(|entry| (entry.api.object.clone(), entry.api.status.clone(), entry.api.status_code))
.collect::<Vec<_>>();
assert_eq!(
audits,
[
(Some("object".to_owned()), Some("success".to_owned()), Some(200)),
(Some("missing".to_owned()), Some("failure".to_owned()), Some(500)),
],
);
assert!(probe.events.lock().unwrap().is_empty());
server.shutdown().await;
})
})
}
#[test]
#[serial_test::serial]
fn legacy_get_object_attributes_completion() {
get_object_attributes_completion(None);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_get_object_attributes_completion() {
get_object_attributes_completion(Some("gateway"));
}
+91 -97
View File
@@ -14,18 +14,20 @@
//! CopyObject path.
use super::copy_input::{CopyObjectWriteInput, CopyObjectWriteOutput};
use super::*;
use crate::app::storage_api::object_usecase::object_utils::to_s3s_xml_etag;
use crate::app::storage_api::s3::dto::{
ChecksumType, CopyObjectInput, CopyObjectOutput, CopyObjectResult, CopySource, ETag, MetadataDirective, StorageClass,
TaggingDirective, Timestamp,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::RequestEnvelope;
use crate::app::storage_api::object_usecase::access::CopySourceObject;
use crate::app::storage_api::s3::dto::Timestamp;
#[cfg(test)]
use crate::app::storage_api::s3::dto::{CopyObjectInput, CopySource, MetadataDirective, StorageClass};
use futures::FutureExt;
use rustfs_gateway_types::CopySourceValue;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
use rustfs_utils::path::fold_object_name_path_separators;
use tracing::Instrument;
use crate::auth::{VerifiedPresignedRequest, reject_presigned_put_max_content_length_for_other_operation};
use crate::auth::reject_presigned_put_max_content_length_for_other_operation;
use crate::error::ServerSideSourceReadError;
struct CopySourceReadStream<R> {
@@ -131,11 +133,12 @@ where
}
impl DefaultObjectUsecase {
pub fn execute_copy_object(
pub(crate) fn execute_copy_object_owned(
&self,
req: S3Request<CopyObjectInput>,
) -> impl std::future::Future<Output = S3Result<S3Response<CopyObjectOutput>>> + Send + Unpin + '_ {
let body = self.execute_copy_object_body_boxed(req);
envelope: RequestEnvelope,
input: CopyObjectWriteInput,
) -> impl std::future::Future<Output = S3Result<CopyObjectWriteOutput>> + Send + Unpin + '_ {
let body = self.execute_copy_object_body_boxed(envelope, input);
futures::future::lazy(move |_| body.instrument(tracing::debug_span!("execute_copy_object"))).flatten()
}
@@ -143,24 +146,30 @@ impl DefaultObjectUsecase {
#[inline(never)]
fn execute_copy_object_body_boxed(
&self,
req: S3Request<CopyObjectInput>,
) -> impl std::future::Future<Output = S3Result<S3Response<CopyObjectOutput>>> + Send + Unpin + '_ {
Box::pin(self.execute_copy_object_body(req))
envelope: RequestEnvelope,
input: CopyObjectWriteInput,
) -> impl std::future::Future<Output = S3Result<CopyObjectWriteOutput>> + Send + Unpin + '_ {
Box::pin(self.execute_copy_object_body(envelope, input))
}
async fn execute_copy_object_body(&self, mut req: S3Request<CopyObjectInput>) -> S3Result<S3Response<CopyObjectOutput>> {
async fn execute_copy_object_body(
&self,
mut envelope: RequestEnvelope,
input: CopyObjectWriteInput,
) -> S3Result<CopyObjectWriteOutput> {
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedPresignedRequest>().is_some(),
envelope.headers(),
envelope.uri().query(),
envelope.verified_presigned().is_some(),
)?;
reject_non_wildcard_write_if_none_match(req.input.if_none_match.as_ref(), &req.headers)?;
reject_non_wildcard_write_if_none_match_native(input.specific_if_none_match, envelope.headers())?;
if let Some(context) = &self.context {
let _ = context.object_store();
}
let mut helper = OperationHelper::new(&req, EventName::ObjectCreatedCopy, S3Operation::CopyObject);
let CopyObjectInput {
let mut helper =
OperationHelper::from_view(&envelope.audit_view(), EventName::ObjectCreatedCopy, S3Operation::CopyObject);
let CopyObjectWriteInput {
copy_source,
bucket,
key,
@@ -192,28 +201,27 @@ impl DefaultObjectUsecase {
storage_class,
checksum_algorithm,
..
} = req.input.clone();
} = input;
let requested_checksum_type = checksum_algorithm
.as_ref()
.map(|algorithm| rustfs_rio::ChecksumType::from_string(algorithm.as_str()));
if requested_checksum_type.is_some_and(|checksum_type| !checksum_type.is_set()) {
return Err(s3_error!(InvalidArgument, "Unsupported checksum algorithm"));
}
let (src_bucket, src_key, version_id) = match copy_source {
CopySource::AccessPoint { .. } => return Err(s3_error!(NotImplemented)),
CopySource::Outpost { .. } => return Err(s3_error!(NotImplemented)),
CopySource::Bucket {
ref bucket,
ref key,
version_id,
} => (bucket.to_string(), key.to_string(), version_id.map(|v| v.to_string())),
let Some(CopySourceValue::Bucket {
bucket: src_bucket,
key: src_key,
version_id: requested_src_version_id,
}) = copy_source
else {
return Err(s3_error!(NotImplemented));
};
// Normalize the copy-source version id like GET/HEAD do: trim, treat "null" as the
// nil UUID, and reject malformed ids up front (issue #4238).
let version_id = match version_id {
Some(v) => {
let trimmed = v.trim();
let version_id = match requested_src_version_id.as_deref() {
Some(version) => {
let trimmed = version.trim();
if trimmed.eq_ignore_ascii_case("null") {
Some(Uuid::nil().to_string())
} else if Uuid::parse_str(trimmed).is_ok() {
@@ -230,9 +238,9 @@ impl DefaultObjectUsecase {
{
return Err(s3_error!(InvalidStorageClass));
}
let ssekms_context = extract_ssekms_context_from_headers(&req.headers)?;
let ssekms_context = extract_ssekms_context_from_headers(envelope.headers())?;
validate_sse_headers_for_write(
requested_sse.as_ref().map(|value| value.as_str()),
requested_sse.as_deref(),
requested_kms_key_id.as_ref(),
ssekms_context.as_ref(),
sse_customer_algorithm.as_ref(),
@@ -247,9 +255,9 @@ impl DefaultObjectUsecase {
validate_new_object_key(&key, "COPY (dest)")?;
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &key).await?;
validate_table_catalog_object_mutation(&bucket, &key).await?;
let replaces_metadata = match metadata_directive.as_ref().map(|directive| directive.as_str()) {
None | Some(MetadataDirective::COPY) => false,
Some(MetadataDirective::REPLACE) => true,
let replaces_metadata = match metadata_directive.as_deref() {
None | Some("COPY") => false,
Some("REPLACE") => true,
Some(_) => {
return Err(S3Error::with_message(
S3ErrorCode::InvalidArgument,
@@ -265,7 +273,7 @@ impl DefaultObjectUsecase {
// s3s parsed `input.metadata` before the query hoist ran, so a
// hoisted `x-amz-meta-*` is only visible on the headers now.
super::super::storage_api::object_usecase::options::extract_metadata_from_mime(
&req.headers,
envelope.headers(),
&mut replacement_metadata,
);
apply_standard_object_metadata(
@@ -289,11 +297,11 @@ impl DefaultObjectUsecase {
// where none of these apply (issue #4238).
let replacement_tags = crate::app::storage_api::object_usecase::s3_api::tagging::resolve_copy_object_tags(
tagging.as_deref(),
tagging_directive.as_ref(),
tagging_directive.as_deref(),
)?;
if !replaces_metadata
&& tagging_directive.as_ref().map(TaggingDirective::as_str) != Some(TaggingDirective::REPLACE)
&& tagging_directive.as_deref() != Some("REPLACE")
&& storage_class.is_none()
&& version_id.is_none()
&& src_bucket == bucket
@@ -310,7 +318,7 @@ impl DefaultObjectUsecase {
let src_versioned_enabled = BucketVersioningSys::prefix_enabled(&src_bucket, &src_key).await;
let src_version_suspended = BucketVersioningSys::prefix_suspended(&src_bucket, &src_key).await;
let mut src_opts = copy_src_opts(&src_bucket, &src_key, &req.headers).map_err(ApiError::from)?;
let mut src_opts = copy_src_opts(&src_bucket, &src_key, envelope.headers()).map_err(ApiError::from)?;
src_opts.version_id = version_id.clone();
src_opts.versioned = src_versioned_enabled;
@@ -322,26 +330,26 @@ impl DefaultObjectUsecase {
version_suspended: src_opts.version_suspended,
..Default::default()
};
apply_copy_source_bucket_generation_guard(&req, &src_bucket, &mut src_get_opts)?;
apply_copy_source_bucket_generation_guard(&envelope, &src_bucket, &mut src_get_opts)?;
let mut dst_opts = copy_dst_opts_with_replication_authorization(
&bucket,
&key,
dest_version_id.clone(),
&req.headers,
envelope.headers(),
HashMap::new(),
replication_request_authorized(&req),
replication_request_authorized(&envelope),
)
.await
.map_err(ApiError::from)?;
apply_bucket_generation_guard(&req, &bucket, &mut dst_opts)?;
apply_bucket_generation_guard(&envelope, &bucket, &mut dst_opts)?;
let cp_src_dst_same = path_join_buf(&[&src_bucket, &src_key]) == path_join_buf(&[&bucket, &key]);
let backslash_is_separator = cfg!(target_os = "windows");
let dst_lock_object = copy_namespace_lock_object(&key, backslash_is_separator);
let self_copy_lock = cp_src_dst_same
|| (src_bucket == bucket && copy_namespace_lock_object(&src_key, backslash_is_separator) == dst_lock_object);
let expected_current_version_id = expected_current_version_id(&req.headers)?;
let expected_current_version_id = expected_current_version_id(envelope.headers())?;
if expected_current_version_id.is_some()
&& (!cp_src_dst_same || version_id.is_none() || dest_version_id.is_some() || !dst_opts.versioned)
{
@@ -423,7 +431,7 @@ impl DefaultObjectUsecase {
&& object_lock_checks_required_for_state(&object_lock_config_state));
let previous_current_sizes: Option<Option<(u64, Result<u64, StorageError>)>> = if copy_prelookup_required {
let mut current_opts: ObjectOptions = internal_object_info_lookup_opts(
get_opts(&bucket, &key, dest_version_id.clone(), None, &req.headers)
get_opts(&bucket, &key, dest_version_id.clone(), None, envelope.headers())
.await
.map_err(ApiError::from)?,
);
@@ -456,7 +464,7 @@ impl DefaultObjectUsecase {
let (mut effective_sse, mut effective_kms_key_id) = resolve_bucket_default_sse(
bucket_sse_config.as_ref().map(|(config, _)| config),
requested_sse.map(|value| value.as_str().to_owned()),
requested_sse,
requested_kms_key_id,
has_explicit_ssec,
);
@@ -467,7 +475,7 @@ impl DefaultObjectUsecase {
copy_source_sse_customer_key_md5.as_ref(),
);
let copy_principal = SseKmsPrincipal::from_request(&req);
let copy_principal = SseKmsPrincipal::from_request(&envelope);
if source_bucket_lifecycle_guard.is_lock_lost() {
return Err(ApiError::from(StorageError::NamespaceLockQuorumUnavailable {
@@ -493,9 +501,16 @@ impl DefaultObjectUsecase {
let _source_cancellation_guard = source_cancellation.clone().drop_guard();
let mut src_info = gr.object_info.clone();
let copies_tags = replacement_tags.is_none();
let effective_tags = replacement_tags.unwrap_or_else(|| (*src_info.user_tags).clone());
crate::app::storage_api::object_usecase::access::authorize_copy_object_tags(
&mut req,
&mut envelope,
CopySourceObject {
bucket: &src_bucket,
key: &src_key,
version_id: requested_src_version_id.as_deref(),
},
copies_tags,
&src_info.user_tags,
&effective_tags,
)
@@ -524,25 +539,13 @@ impl DefaultObjectUsecase {
});
// Validate copy source conditions
if let Some(if_match) = copy_source_if_match {
if let Some(ref etag) = src_info.etag {
if let Some(strong_etag) = if_match.into_etag() {
if ETag::Strong(etag.clone()) != strong_etag {
return Err(s3_error!(PreconditionFailed));
}
} else {
// Weak ETag or Any (*) in If-Match should fail per RFC 9110
return Err(s3_error!(PreconditionFailed));
}
} else {
return Err(s3_error!(PreconditionFailed));
}
if let Some(condition) = copy_source_if_match
&& !condition.strong().is_some_and(|tag| Some(tag) == src_info.etag.as_deref())
{
return Err(s3_error!(PreconditionFailed));
}
if let Some(if_none_match) = copy_source_if_none_match
&& let Some(ref etag) = src_info.etag
&& let Some(strong_etag) = if_none_match.into_etag()
&& ETag::Strong(etag.clone()) == strong_etag
if let Some(condition) = copy_source_if_none_match
&& condition.strong().is_some_and(|tag| Some(tag) == src_info.etag.as_deref())
{
return Err(s3_error!(PreconditionFailed));
}
@@ -589,10 +592,7 @@ impl DefaultObjectUsecase {
strip_managed_encryption_metadata(&mut user_defined);
remove_str(&mut user_defined, SUFFIX_PLAINTEXT_CHECKSUM);
let destination_storage_class = storage_class
.as_ref()
.map(StorageClass::as_str)
.unwrap_or(storageclass::STANDARD);
let destination_storage_class = storage_class.as_deref().unwrap_or(storageclass::STANDARD);
src_info.storage_class = Some(destination_storage_class.to_string());
let actual_size = src_info.get_actual_size().map_err(ApiError::from)?;
@@ -601,7 +601,7 @@ impl DefaultObjectUsecase {
let mut compress_metadata = HashMap::new();
let should_compress = is_disk_compressible(&req.headers, &key) && actual_size > MIN_DISK_COMPRESSIBLE_SIZE as i64;
let should_compress = is_disk_compressible(envelope.headers(), &key) && actual_size > MIN_DISK_COMPRESSIBLE_SIZE as i64;
if should_compress {
insert_str(
@@ -651,7 +651,7 @@ impl DefaultObjectUsecase {
let has_explicit_object_lock_retention = object_lock_mode.is_some() || object_lock_retain_until_date.is_some();
remove_object_lock_metadata_for_copy(&mut user_defined);
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata(
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata_native(
&bucket,
&object_lock_config_state,
object_lock_legal_hold_status,
@@ -795,7 +795,7 @@ impl DefaultObjectUsecase {
let has_bucket_metadata = self.bucket_metadata_sys().is_some();
let cache_adapter = self.object_data_cache();
let _ = invalidate_object_data_cache_before_mutation(&cache_adapter, &bucket, &key).await;
let table_data_plane_publication_guards = retained_table_data_plane_publication_guards(&req);
let table_data_plane_publication_guards = retained_table_data_plane_publication_guards(&envelope);
let copy_commit = spawn_traced_join({
let store = Arc::clone(&store);
@@ -878,7 +878,7 @@ impl DefaultObjectUsecase {
// Report the destination object's checksum in the response, decoded the same way GetObject
// / HeadObject do so the value is identical to a later checksum-mode HEAD/GET (#4996).
let response_checksums = oi
.decrypt_checksums(0, &req.headers)
.decrypt_checksums(0, envelope.headers())
.map(|(pairs, is_multipart)| classify_response_checksums(pairs, is_multipart))
.unwrap_or_default();
@@ -899,9 +899,15 @@ impl DefaultObjectUsecase {
_ => {}
}
}
let copy_object_result = CopyObjectResult {
e_tag: oi.etag.as_ref().map(|etag| to_s3s_xml_etag(etag)),
last_modified: oi.mod_time.map(Timestamp::from),
let output = CopyObjectWriteOutput {
e_tag: oi.etag,
last_modified: oi.mod_time,
copy_source_version_id,
version_id: dest_version,
server_side_encryption: effective_sse,
ssekms_key_id: effective_kms_key_id,
sse_customer_algorithm,
sse_customer_key_md5,
checksum_crc32: response_checksums.crc32,
checksum_crc32c: response_checksums.crc32c,
checksum_sha1: response_checksums.sha1,
@@ -912,25 +918,12 @@ impl DefaultObjectUsecase {
checksum_xxhash3,
checksum_xxhash64,
checksum_xxhash128,
checksum_type: response_checksums.checksum_type.map(ChecksumType::from),
checksum_type: response_checksums.checksum_type,
};
let output = CopyObjectOutput {
copy_object_result: Some(copy_object_result),
copy_source_version_id,
server_side_encryption: effective_sse.map(Into::into),
ssekms_key_id: effective_kms_key_id,
sse_customer_algorithm,
sse_customer_key_md5,
version_id: dest_version,
..Default::default()
};
let version_id = req.input.version_id.clone().unwrap_or_default();
helper = helper.object(object_info).version_id(version_id);
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
helper = helper.object(object_info).version_id(dest_version_id.unwrap_or_default());
let result = Ok(output);
let _ = helper.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
result
}
}
@@ -938,6 +931,7 @@ impl DefaultObjectUsecase {
#[cfg(test)]
mod tests {
use super::*;
use crate::app::storage_api::s3::S3ErrorCode;
use crate::app::storage_api::s3::dto::ServerSideEncryption;
use http::{HeaderValue, Method};
use rustfs_gateway_types::persistence::{
+93
View File
@@ -0,0 +1,93 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Owned input and output for the CopyObject storage path.
use rustfs_gateway_types::CopySourceValue;
use std::collections::HashMap;
use time::OffsetDateTime;
/// Comparison strength for decoded source conditions.
pub(crate) enum CopySourceCondition {
Any,
Strong(String),
Weak,
}
impl CopySourceCondition {
pub(crate) fn strong(&self) -> Option<&str> {
match self {
Self::Strong(value) => Some(value),
Self::Any | Self::Weak => None,
}
}
}
// Deliberately no Debug: these fields may contain customer encryption keys.
pub(crate) struct CopyObjectWriteInput {
pub(crate) bucket: String,
pub(crate) key: String,
/// Only bucket-path sources are supported.
pub(crate) copy_source: Option<CopySourceValue>,
pub(crate) version_id: Option<String>,
pub(crate) specific_if_none_match: bool,
pub(crate) server_side_encryption: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
pub(crate) copy_source_sse_customer_algorithm: Option<String>,
pub(crate) copy_source_sse_customer_key: Option<String>,
pub(crate) copy_source_sse_customer_key_md5: Option<String>,
pub(crate) metadata_directive: Option<String>,
pub(crate) metadata: Option<HashMap<String, String>>,
pub(crate) tagging: Option<String>,
pub(crate) tagging_directive: Option<String>,
pub(crate) copy_source_if_match: Option<CopySourceCondition>,
pub(crate) copy_source_if_none_match: Option<CopySourceCondition>,
pub(crate) cache_control: Option<String>,
pub(crate) content_disposition: Option<String>,
pub(crate) content_encoding: Option<String>,
pub(crate) content_language: Option<String>,
pub(crate) content_type: Option<String>,
pub(crate) expires: Option<String>,
pub(crate) website_redirect_location: Option<String>,
pub(crate) object_lock_legal_hold_status: Option<String>,
pub(crate) object_lock_mode: Option<String>,
pub(crate) object_lock_retain_until_date: Option<OffsetDateTime>,
pub(crate) storage_class: Option<String>,
pub(crate) checksum_algorithm: Option<String>,
}
pub(crate) struct CopyObjectWriteOutput {
pub(crate) e_tag: Option<String>,
pub(crate) last_modified: Option<OffsetDateTime>,
pub(crate) copy_source_version_id: Option<String>,
pub(crate) version_id: Option<String>,
pub(crate) server_side_encryption: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
pub(crate) checksum_crc32: Option<String>,
pub(crate) checksum_crc32c: Option<String>,
pub(crate) checksum_sha1: Option<String>,
pub(crate) checksum_sha256: Option<String>,
pub(crate) checksum_crc64nvme: Option<String>,
pub(crate) checksum_md5: Option<String>,
pub(crate) checksum_sha512: Option<String>,
pub(crate) checksum_xxhash3: Option<String>,
pub(crate) checksum_xxhash64: Option<String>,
pub(crate) checksum_xxhash128: Option<String>,
pub(crate) checksum_type: Option<String>,
}
+145 -122
View File
@@ -18,10 +18,14 @@ use super::shared::{LOG_COMPONENT_APP, LOG_SUBSYSTEM_OBJECT};
use super::*;
use crate::app::RequestEnvelope;
use crate::app::storage_api::object_usecase::DeleteObjectCondition;
use crate::app::storage_api::s3::dto::ObjectIdentifier;
use crate::app::storage_api::s3::dto::{DeleteObjectsInput, DeleteObjectsOutput, DeletedObject};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::storage_api::object_usecase::access::AuthorizationRequest as _;
#[cfg(test)]
use crate::app::storage_api::s3::dto::{DeleteObjectsInput, DeleteObjectsOutput};
use crate::app::storage_api::s3::{S3ErrorCode, s3_error};
#[cfg(test)]
use crate::app::storage_api::s3::{S3Request, S3Response};
use rustfs_gateway_types::dto as gateway_dto;
use rustfs_s3_types::S3Result;
use std::borrow::Cow;
use std::collections::{BTreeMap, HashSet};
@@ -100,11 +104,10 @@ fn normalize_delete_object_key(key: &str) -> Cow<'_, str> {
}
fn successful_delete_audit_objects(
delete: &s3s::dto::Delete,
objects: &[DeleteBatchEntry],
successful_results: impl IntoIterator<Item = bool>,
) -> Vec<AuditObjectVersion> {
delete
.objects
objects
.iter()
.zip(successful_results)
.filter(|(_, successful)| *successful)
@@ -131,7 +134,7 @@ fn normalize_delete_objects_version_id(
}
}
fn parse_delete_objects_condition(object: &ObjectIdentifier) -> Result<Option<DeleteObjectCondition>, s3s::dto::Error> {
fn parse_delete_objects_condition(object: &DeleteBatchEntry) -> Result<Option<DeleteObjectCondition>, DeleteBatchError> {
let Some(etag) = object.e_tag.as_ref() else {
return Ok(None);
};
@@ -140,15 +143,15 @@ fn parse_delete_objects_condition(object: &ObjectIdentifier) -> Result<Option<De
S3ErrorCode::InvalidRequest,
"An ETag condition cannot be combined with an explicit VersionId".to_string(),
)
} else if let Some(etag) = etag.as_strong() {
match DeleteObjectCondition::parse(etag) {
} else if !etag.weak {
match DeleteObjectCondition::parse(&etag.value) {
Ok(condition) => return Ok(Some(condition)),
Err(error) => (S3ErrorCode::InvalidArgument, error.to_string()),
}
} else {
(S3ErrorCode::InvalidArgument, "A delete ETag condition must use a strong ETag".to_string())
};
Err(s3s::dto::Error {
Err(DeleteBatchError {
code: Some(code.as_str().to_string()),
key: Some(object.key.clone()),
message: Some(message),
@@ -341,7 +344,7 @@ async fn replica_delete_fanout_state(
replica_delete_fanout_state_from_config(config, replication_source, version_id, &site_peer_deployment_ids)
}
fn validate_undo_delete_version(expected: Option<&str>, requested: Option<&str>) -> S3Result<()> {
fn validate_undo_delete_version(expected: Option<&str>, requested: Option<&str>) -> crate::app::storage_api::s3::S3Result<()> {
if expected.is_some() && expected != requested {
return Err(s3_error!(PreconditionFailed));
}
@@ -485,12 +488,12 @@ fn project_delete_objects_pre_stat_error(
object: ObjectToDelete,
synthetic_version_id: bool,
error: ApiError,
) -> S3Result<s3s::dto::Error> {
) -> S3Result<DeleteBatchError> {
// Bucket loss invalidates the shared request, not just one object.
if error.code == S3ErrorCode::NoSuchBucket {
return Err(error.into());
}
Ok(s3s::dto::Error {
Ok(DeleteBatchError {
code: Some(error.code.as_str().to_string()),
key: Some(object.object_name),
message: Some(error.message),
@@ -524,12 +527,12 @@ fn reduce_delete_objects_result<'a>(
deleted: &'a StorageDeletedObject,
error: Option<&EcstoreError>,
synthetic_version_id: bool,
) -> Result<&'a StorageDeletedObject, s3s::dto::Error> {
) -> Result<&'a StorageDeletedObject, DeleteBatchError> {
match delete_objects_reported_storage_error(object, error) {
None => Ok(deleted),
Some(error) => {
let api_error = ApiError::from(error.clone());
Err(s3s::dto::Error {
Err(DeleteBatchError {
code: Some(api_error.code.as_str().to_string()),
key: Some(object.object_name.clone()),
message: Some(api_error.message),
@@ -582,50 +585,54 @@ fn delete_objects_storage_error_counts(
}
impl DefaultObjectUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_delete_objects(&self, req: S3Request<DeleteObjectsInput>) -> S3Result<S3Response<DeleteObjectsOutput>> {
#[instrument(level = "debug", skip(self, envelope, input))]
pub(crate) async fn execute_delete_objects(
&self,
envelope: RequestEnvelope,
input: DeleteBatchInput,
) -> S3Result<DeleteBatchOutput> {
let owner = self.clone();
run_owned_mutation(async move { Box::pin(owner.execute_delete_objects_owned(req)).await }).await
run_owned_mutation(async move { Box::pin(owner.execute_delete_objects_owned(envelope, input)).await }).await
}
async fn execute_delete_objects_owned(
&self,
mut req: S3Request<DeleteObjectsInput>,
) -> S3Result<S3Response<DeleteObjectsOutput>> {
mut envelope: RequestEnvelope,
input: DeleteBatchInput,
) -> S3Result<DeleteBatchOutput> {
if let Some(context) = &self.context {
let _ = context.object_store();
}
let helper = OperationHelper::new(&req, EventName::ObjectRemovedDelete, S3Operation::DeleteObjects).suppress_event();
let request_context = helper.request_context_or_from_request(&req);
let (bucket, delete) = {
let bucket = req.input.bucket.clone();
let delete = req.input.delete.clone();
(bucket, delete)
};
let view = envelope.audit_view();
let helper =
OperationHelper::from_view(&view, EventName::ObjectRemovedDelete, S3Operation::DeleteObjects).suppress_event();
let request_context = helper.request_context_or_from_view(&view);
// Both edges carry LastModifiedTime and Size. The batch engine only evaluates ETag.
let DeleteBatchInput { bucket, objects, quiet } = input;
if delete.objects.is_empty() || delete.objects.len() > 1000 {
return Err(S3Error::with_message(
S3ErrorCode::InvalidArgument,
"No objects to delete or too many objects to delete".to_string(),
if objects.is_empty() || objects.len() > 1000 {
return Err(rustfs_s3_types::s3_error!(
InvalidArgument,
"No objects to delete or too many objects to delete"
));
}
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
return Err(rustfs_s3_types::s3_error!(InternalError, "Not init"));
};
// Capture the bucket generation before per-object authorization, but
// do not expose a bucket-state error unless at least one object is authorized.
let bucket_generation = load_bucket_generation_from_store(store.as_ref(), &req, &bucket).await;
let bucket_generation = load_bucket_generation_from_store(store.as_ref(), &envelope, &bucket).await;
#[cfg(test)]
wait_for_delete_objects_generation_test_hook(&bucket).await;
let bypass_governance = has_bypass_governance_header(&req.headers);
let bypass_governance = has_bypass_governance_header(envelope.headers());
#[derive(Default, Clone)]
struct DeleteResult {
delete_object: Option<StorageDeletedObject>,
error: Option<s3s::dto::Error>,
error: Option<DeleteBatchError>,
emit_side_effects: bool,
synthetic_version_id: bool,
version_suspended: bool,
@@ -636,29 +643,40 @@ impl DefaultObjectUsecase {
requested_version_id: Option<Uuid>,
}
let mut delete_results = vec![DeleteResult::default(); delete.objects.len()];
let mut delete_results = vec![DeleteResult::default(); objects.len()];
struct AuthorizedDelete {
idx: usize,
object: ObjectToDelete,
}
let mut authorized_deletes = Vec::with_capacity(delete.objects.len());
let mut authorized_deletes = Vec::with_capacity(objects.len());
// Issue #5740: keep the first per-key denial of this bulk request at
// warn and demote the rest to debug, so a denied 1000-key DeleteObjects
// cannot flood the log.
let mut bulk_denial_logged = false;
// Per-key denials are audited once per denied action, in aggregate, after the loop.
let mut bulk_denials = crate::storage::access::BulkDenialAudit::start(&mut req)?;
for (idx, obj_id) in delete.objects.iter().enumerate() {
let mut bulk_denials = crate::storage::access::BulkDenialAudit::start_owned(&mut envelope)?;
for (idx, obj_id) in objects.iter().enumerate() {
// Authorization and every storage side effect must address the same
// normalized object; retain the request key only for response entries.
let key = normalize_delete_object_key(&obj_id.key);
if let Err(err) = validate_object_key(&key, "DeleteObjects") {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some(err.code().as_str().to_string()),
// A rejected wire key has no authorization resource. Its normalized text cannot become a target.
let key_error = if obj_id.key_too_long {
Some((
S3ErrorCode::InvalidArgument,
ApiError::error_code_to_message(&S3ErrorCode::InvalidArgument),
))
} else {
validate_object_key(&key, "DeleteObjects")
.err()
.map(|error| (error.code().clone(), error.to_string()))
};
if let Some((code, message)) = key_error {
delete_results[idx].error = Some(DeleteBatchError {
code: Some(code.as_str().to_string()),
key: Some(obj_id.key.clone()),
message: Some(err.to_string()),
message: Some(message),
version_id: obj_id.version_id.clone(),
});
continue;
@@ -674,7 +692,7 @@ impl DefaultObjectUsecase {
let (version_id, version_uuid) = match normalize_delete_objects_version_id(raw_version_id.clone()) {
Ok(parsed) => parsed,
Err(err) => {
delete_results[idx].error = Some(s3s::dto::Error {
delete_results[idx].error = Some(DeleteBatchError {
code: Some("NoSuchVersion".to_string()),
key: Some(obj_id.key.clone()),
message: Some(err),
@@ -685,29 +703,29 @@ impl DefaultObjectUsecase {
};
{
let req_info = req_info_mut(&mut req)?;
let req_info = req_info_mut(&mut envelope)?;
req_info.bucket = Some(bucket.clone());
req_info.object = Some(key.to_string());
req_info.version_id = version_id.clone();
}
let mut denied_action = delete_object_authorize_action(version_id.as_deref());
let mut auth_res = authorize_request(&mut req, denied_action).await;
let mut auth_res = authorize_request(&mut envelope, denied_action).await;
if auth_res.is_ok()
&& delete_condition
.as_ref()
.is_some_and(DeleteObjectCondition::requires_read_permission)
{
denied_action = Action::S3Action(S3Action::GetObjectAction);
auth_res = authorize_request(&mut req, denied_action).await;
auth_res = authorize_request(&mut envelope, denied_action).await;
}
if auth_res.is_err() {
bulk_denials.denied(denied_action);
if !bulk_denial_logged {
bulk_denial_logged = true;
req_info_mut(&mut req)?.suppress_denial_log = true;
req_info_mut(&mut envelope)?.suppress_denial_log = true;
}
delete_results[idx].error = Some(s3s::dto::Error {
delete_results[idx].error = Some(DeleteBatchError {
code: Some("AccessDenied".to_string()),
key: Some(obj_id.key.clone()),
message: Some("Access Denied".to_string()),
@@ -717,14 +735,15 @@ impl DefaultObjectUsecase {
}
if bypass_governance {
let auth_res = authorize_request(&mut req, Action::S3Action(S3Action::BypassGovernanceRetentionAction)).await;
let auth_res =
authorize_request(&mut envelope, Action::S3Action(S3Action::BypassGovernanceRetentionAction)).await;
if auth_res.is_err() {
bulk_denials.denied(Action::S3Action(S3Action::BypassGovernanceRetentionAction));
if !bulk_denial_logged {
bulk_denial_logged = true;
req_info_mut(&mut req)?.suppress_denial_log = true;
req_info_mut(&mut envelope)?.suppress_denial_log = true;
}
delete_results[idx].error = Some(s3s::dto::Error {
delete_results[idx].error = Some(DeleteBatchError {
code: Some("AccessDenied".to_string()),
key: Some(obj_id.key.clone()),
message: Some("Access Denied".to_string()),
@@ -735,7 +754,7 @@ impl DefaultObjectUsecase {
}
if let Err(err) = validate_table_catalog_object_mutation(&bucket, &key).await {
delete_results[idx].error = Some(s3s::dto::Error {
delete_results[idx].error = Some(DeleteBatchError {
code: Some("InvalidRequest".to_string()),
key: Some(obj_id.key.clone()),
message: Some(err.to_string()),
@@ -758,19 +777,18 @@ impl DefaultObjectUsecase {
authorized_deletes.push(AuthorizedDelete { idx, object });
}
bulk_denials.finish(&mut req)?;
bulk_denials.finish_owned(&mut envelope, RequestEnvelope::audit_view)?;
if authorized_deletes.is_empty() {
let output = DeleteObjectsOutput {
deleted: (!delete.quiet.unwrap_or_default()).then(Vec::new),
let output = DeleteBatchOutput {
deleted: (!quiet.unwrap_or_default()).then(Vec::new),
errors: Some(delete_results.into_iter().filter_map(|result| result.error).collect()),
..Default::default()
};
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
let result = Ok(output);
let _ = helper.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
return result;
}
req.extensions.insert(bucket_generation?);
envelope.install_bucket_generation(bucket_generation?);
let bucket_lock_enabled = object_lock_checks_required(&bucket).await;
let delete_config_snapshot = Arc::new(
@@ -795,7 +813,7 @@ impl DefaultObjectUsecase {
// DeleteObjects names each key explicitly. The force-delete header is a
// DeleteObject/DeleteBucket extension and must not change this batch,
// including by rejecting a non-boolean value.
let mut batch_headers = req.headers.clone();
let mut batch_headers = envelope.headers().clone();
batch_headers.remove("x-rustfs-force-delete");
batch_headers.remove("x-minio-force-delete");
let mut prepared_deletes: Vec<PreparedDelete> = Vec::with_capacity(authorized_deletes.len());
@@ -913,7 +931,7 @@ impl DefaultObjectUsecase {
object_lock_delete: Some(StorageObjectLockDeleteOptions { bypass_governance }),
..Default::default()
};
apply_bucket_generation_guard(&req, &bucket, &mut storage_delete_opts)?;
apply_bucket_generation_guard(&envelope, &bucket, &mut storage_delete_opts)?;
let storage_started = std::time::Instant::now();
let (dobjs, errs, accounting) = store
.delete_objects_with_tier_delete_journal_and_accounting(&bucket, object_to_delete.clone(), storage_delete_opts)
@@ -952,8 +970,8 @@ impl DefaultObjectUsecase {
super::super::storage_api::object_usecase::helper::pause_committed_mutation_for_test(&bucket).await;
let _manager = get_concurrency_manager();
if !errs.is_empty() && errs.iter().all(|err| err.as_ref().is_some_and(is_err_bucket_not_found)) {
let result = Err(S3Error::with_message(S3ErrorCode::NoSuchBucket, "Bucket not found".to_string()));
let _ = helper.complete(&result);
let result = Err(rustfs_s3_types::s3_error!(NoSuchBucket, "Bucket not found"));
let _ = helper.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
return result;
}
@@ -1001,19 +1019,19 @@ impl DefaultObjectUsecase {
}
}
let deleted = (!delete.quiet.unwrap_or_default()).then(|| {
let deleted = (!quiet.unwrap_or_default()).then(|| {
delete_results
.iter()
.enumerate()
.filter_map(|(idx, result)| result.delete_object.as_ref().map(|object| (idx, result, object)))
.map(|(idx, result, object)| DeletedObject {
.map(|(idx, result, object)| DeleteBatchSuccess {
delete_marker: { if object.delete_marker { Some(true) } else { None } },
delete_marker_version_id: delete_marker_response_version_id(
object.delete_marker_version_id,
object.delete_marker && result.version_suspended && result.requested_version_id.is_none(),
result.synthetic_version_id,
),
key: Some(delete.objects[idx].key.clone()),
key: Some(objects[idx].key.clone()),
version_id: delete_entry_response_version_id(
object.version_id,
object.delete_marker,
@@ -1034,18 +1052,17 @@ impl DefaultObjectUsecase {
.enumerate()
.filter_map(|(idx, result)| {
let mut error = result.error.clone()?;
error.key = Some(delete.objects[idx].key.clone());
error.key = Some(objects[idx].key.clone());
Some(error)
})
.collect::<Vec<s3s::dto::Error>>();
let output = DeleteObjectsOutput {
.collect::<Vec<_>>();
let output = DeleteBatchOutput {
deleted,
errors: Some(errors),
..Default::default()
};
let helper = if helper.wants_audit_object_info() {
let audit_objects =
successful_delete_audit_objects(&delete, delete_results.iter().map(|result| result.delete_object.is_some()));
successful_delete_audit_objects(&objects, delete_results.iter().map(|result| result.delete_object.is_some()));
helper.audit_objects(audit_objects)
} else {
helper
@@ -1074,11 +1091,10 @@ impl DefaultObjectUsecase {
let _ = replication_task.await;
}
let req_headers = req.headers.clone();
let req_headers = envelope.headers().clone();
let notify = current_notify_interface_for_context(self.context.as_deref());
let req_params = rustfs_targets::extract_params_header(&req_headers);
let resp_elements =
build_event_resp_elements(&S3Response::new(DeleteObjectsOutput::default()), &request_context.request_id);
let resp_elements = build_owned_event_resp_elements(&HeaderMap::new(), &request_context.request_id);
let deleted_any = delete_results.iter().any(|result| result.delete_object.is_some());
let delete_producers = delete_results
.iter()
@@ -1143,8 +1159,8 @@ impl DefaultObjectUsecase {
}
});
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
let result = Ok(output);
let _ = helper.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
if deleted_any {
rustfs_scanner::record_dirty_usage_bucket_from_producers(&bucket, delete_producers);
}
@@ -1600,12 +1616,20 @@ impl DefaultObjectUsecase {
#[cfg(test)]
impl DefaultObjectUsecase {
pub(crate) async fn execute_delete_objects_from_s3s(
&self,
req: S3Request<DeleteObjectsInput>,
) -> crate::app::storage_api::s3::S3Result<S3Response<DeleteObjectsOutput>> {
let envelope = RequestEnvelope::from_s3s(&req);
Ok(S3Response::new(self.execute_delete_objects(envelope, req.input.into()).await?.into()))
}
/// DeleteObject for a request built as the legacy stack receives it, converted the way its
/// edge converts it.
pub(crate) async fn execute_delete_object_from_s3s(
&self,
req: S3Request<crate::app::storage_api::s3::dto::DeleteObjectInput>,
) -> S3Result<gateway_dto::DeleteObjectOutput> {
) -> crate::app::storage_api::s3::S3Result<gateway_dto::DeleteObjectOutput> {
let envelope = RequestEnvelope::from_s3s(&req);
let input = req.input;
Ok(Box::pin(self.execute_delete_object_fields(envelope, input.bucket, input.key, input.version_id)).await?)
@@ -1843,7 +1867,7 @@ mod tests {
is_owner: true,
..Default::default()
});
let output = Box::pin(usecase.execute_delete_objects(req))
let output = Box::pin(usecase.execute_delete_objects_from_s3s(req))
.await
.expect("batch delete")
.output;
@@ -2371,7 +2395,7 @@ mod tests {
is_owner: true,
..Default::default()
});
let response = Box::pin(usecase.execute_delete_objects(req))
let response = Box::pin(usecase.execute_delete_objects_from_s3s(req))
.await
.expect("batch marker delete must reach the authoritative mutation");
assert!(response.output.errors.as_ref().is_none_or(Vec::is_empty), "{:?}", response.output.errors);
@@ -2552,7 +2576,7 @@ mod tests {
is_owner: true,
..Default::default()
});
let response = Box::pin(usecase.execute_delete_objects(req))
let response = Box::pin(usecase.execute_delete_objects_from_s3s(req))
.await
.expect("a pre-stat failure must not abort independent batch entries");
let deleted = response.output.deleted.expect("per-key successes");
@@ -2586,7 +2610,7 @@ mod tests {
is_owner: true,
..Default::default()
});
let response = Box::pin(usecase.execute_delete_objects(req))
let response = Box::pin(usecase.execute_delete_objects_from_s3s(req))
.await
.expect("all pre-stat failures still produce per-key results");
assert!(response.output.deleted.as_ref().is_none_or(Vec::is_empty));
@@ -2654,7 +2678,7 @@ mod tests {
ApiError::from(StorageError::BucketNotFound("missing".to_string())),
)
.expect_err("shared bucket loss remains a request failure");
assert_eq!(error.code(), &S3ErrorCode::NoSuchBucket);
assert_eq!(error.code(), &rustfs_s3_types::S3ErrorCode::NoSuchBucket);
let version = Uuid::new_v4();
for (version_id, synthetic, expected) in [
@@ -2818,7 +2842,7 @@ mod tests {
let req = build_request(input, Method::POST);
let usecase = DefaultObjectUsecase::without_context();
let err = usecase.execute_delete_objects(req).await.unwrap_err();
let err = usecase.execute_delete_objects_from_s3s(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
}
@@ -2840,7 +2864,7 @@ mod tests {
let req = build_request(input, Method::POST);
let usecase = DefaultObjectUsecase::without_context();
let err = usecase.execute_delete_objects(req).await.unwrap_err();
let err = usecase.execute_delete_objects_from_s3s(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
}
@@ -2862,7 +2886,7 @@ mod tests {
let req = build_request(input, Method::POST);
let usecase = DefaultObjectUsecase::without_context();
let err = usecase.execute_delete_objects(req).await.unwrap_err();
let err = usecase.execute_delete_objects_from_s3s(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
assert_eq!(err.message(), Some("Not init"));
}
@@ -2920,7 +2944,7 @@ mod tests {
install_delete_objects_generation_test_hook(bucket.clone(), Arc::clone(&loaded), Arc::clone(&resume));
let usecase = DefaultObjectUsecase::with_context(Some(context));
let delete = tokio::spawn(async move { usecase.execute_delete_objects(req).await });
let delete = tokio::spawn(async move { usecase.execute_delete_objects_from_s3s(req).await });
loaded.wait().await;
store
@@ -3046,13 +3070,8 @@ mod tests {
},
];
let objects = successful_delete_audit_objects(
&Delete {
objects: requested,
quiet: Some(true),
},
[true, false, true],
);
let requested: Vec<_> = requested.into_iter().map(Into::into).collect();
let objects = successful_delete_audit_objects(&requested, [true, false, true]);
assert_eq!(
objects,
@@ -3071,16 +3090,8 @@ mod tests {
..Default::default()
}];
assert!(
successful_delete_audit_objects(
&Delete {
objects: requested,
quiet: None,
},
[false]
)
.is_empty()
);
let requested: Vec<_> = requested.into_iter().map(Into::into).collect();
assert!(successful_delete_audit_objects(&requested, [false]).is_empty());
}
#[test]
@@ -3103,22 +3114,31 @@ mod tests {
key: "key".to_string(),
..Default::default()
};
assert_eq!(parse_delete_objects_condition(&object).expect("missing ETag is unconditional"), None);
assert_eq!(
parse_delete_objects_condition(&object.clone().into()).expect("missing ETag is unconditional"),
None
);
for etag in ["", "\"\"", " \t "] {
let error = parse_delete_objects_condition(&ObjectIdentifier {
e_tag: Some(ETag::Strong(etag.to_string())),
..object.clone()
})
let error = parse_delete_objects_condition(
&ObjectIdentifier {
e_tag: Some(ETag::Strong(etag.to_string())),
..object.clone()
}
.into(),
)
.expect_err("an explicit empty ETag must never become an unconditional delete");
assert_eq!(error.code.as_deref(), Some("InvalidArgument"));
assert_eq!(error.key.as_deref(), Some("key"));
}
for version in ["null", "", "bad-version"] {
let error = parse_delete_objects_condition(&ObjectIdentifier {
e_tag: Some(ETag::Strong("*".to_string())),
version_id: Some(version.to_string()),
..object.clone()
})
let error = parse_delete_objects_condition(
&ObjectIdentifier {
e_tag: Some(ETag::Strong("*".to_string())),
version_id: Some(version.to_string()),
..object.clone()
}
.into(),
)
.expect_err("any explicit VersionId is unsupported with a condition");
assert_eq!(error.code.as_deref(), Some("InvalidRequest"));
assert_eq!(error.version_id.as_deref(), Some(version));
@@ -3159,7 +3179,7 @@ mod tests {
e_tag: Some(ETag::Weak(value.to_string())),
..Default::default()
};
let error = parse_delete_objects_condition(&object)
let error = parse_delete_objects_condition(&object.into())
.expect_err("a weak validator must fail instead of falling back to an unconditional delete");
assert_eq!(error.code.as_deref(), Some("InvalidArgument"));
assert_eq!(error.key.as_deref(), Some("key"));
@@ -3170,7 +3190,7 @@ mod tests {
..Default::default()
};
assert_eq!(
parse_delete_objects_condition(&object).expect("strong validator retains its condition"),
parse_delete_objects_condition(&object.into()).expect("strong validator retains its condition"),
Some(DeleteObjectCondition::MatchEtag("current-etag".to_string()))
);
}
@@ -3398,7 +3418,7 @@ mod tests {
req.extensions.insert(crate::storage::access::ReqInfo::default());
let usecase = DefaultObjectUsecase::from_global();
let response = usecase
.execute_delete_objects(req)
.execute_delete_objects_from_s3s(req)
.await
.expect("mixed batch returns per-entry outcomes");
let deleted = response
@@ -3470,7 +3490,10 @@ mod tests {
.expect("build quiet mixed batch");
let mut quiet_req = build_request(quiet_input, Method::POST);
quiet_req.extensions.insert(crate::storage::access::ReqInfo::default());
let quiet_response = usecase.execute_delete_objects(quiet_req).await.expect("quiet batch succeeds");
let quiet_response = usecase
.execute_delete_objects_from_s3s(quiet_req)
.await
.expect("quiet batch succeeds");
assert!(quiet_response.output.deleted.is_none(), "Quiet hides only successful entries");
let quiet_errors = quiet_response.output.errors.expect("Quiet preserves errors");
assert_eq!(quiet_errors.len(), 1);
@@ -3562,7 +3585,7 @@ mod tests {
let mut request = build_request(input, Method::POST);
request.uri = format!("/{request_bucket}").parse().expect("URI");
request.extensions.insert(crate::storage::access::ReqInfo::default());
usecase.execute_delete_objects(request).await.map(|_| ())
usecase.execute_delete_objects_from_s3s(request).await.map(|_| ())
} else {
let input = DeleteObjectInput::builder()
.bucket(request_bucket.clone())
@@ -3942,7 +3965,7 @@ mod tests {
..Default::default()
});
DefaultObjectUsecase::from_global()
.execute_delete_objects(batch_req)
.execute_delete_objects_from_s3s(batch_req)
.await
.expect("batch compressed delete should succeed");
assert_eq!(
@@ -4322,10 +4345,10 @@ mod bulk_denial_audit_wiring_tests {
.next()
.unwrap();
let started = source
.find("BulkDenialAudit::start(&mut req)")
.find("BulkDenialAudit::start_owned(&mut envelope)")
.expect("per-key denial audit is not suppressed");
let finished = source
.find("bulk_denials.finish(&mut req)")
.find("bulk_denials.finish_owned(&mut envelope, RequestEnvelope::audit_view)")
.expect("the aggregate entry is not emitted");
assert!(started < finished, "the aggregate must be emitted after the per-key checks");
assert_eq!(
@@ -0,0 +1,63 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Owned input and result fields for the batch deletion engine.
//!
//! Keys and versions stay raw so legacy refusals can echo entries that did not pass validation.
#[derive(Debug, Default)]
pub(crate) struct DeleteBatchInput {
pub(crate) bucket: String,
pub(crate) objects: Vec<DeleteBatchEntry>,
pub(crate) quiet: Option<bool>,
}
#[derive(Debug, Default, Clone, PartialEq, Eq)]
pub(crate) struct DeleteBatchEntry {
pub(crate) key: String,
pub(crate) key_too_long: bool,
pub(crate) version_id: Option<String>,
pub(crate) e_tag: Option<DeleteBatchETag>,
/// Seconds since the Unix epoch and nanoseconds within that second.
pub(crate) _last_modified_time: Option<(i64, u32)>,
pub(crate) _size: Option<i64>,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct DeleteBatchETag {
pub(crate) value: String,
pub(crate) weak: bool,
}
#[derive(Debug, Default)]
pub(crate) struct DeleteBatchOutput {
pub(crate) deleted: Option<Vec<DeleteBatchSuccess>>,
pub(crate) errors: Option<Vec<DeleteBatchError>>,
}
#[derive(Debug, Default)]
pub(crate) struct DeleteBatchSuccess {
pub(crate) delete_marker: Option<bool>,
pub(crate) delete_marker_version_id: Option<String>,
pub(crate) key: Option<String>,
pub(crate) version_id: Option<String>,
}
#[derive(Debug, Default, Clone)]
pub(crate) struct DeleteBatchError {
pub(crate) code: Option<String>,
pub(crate) key: Option<String>,
pub(crate) message: Option<String>,
pub(crate) version_id: Option<String>,
}
+2 -272
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! GetObject / GetObjectAttributes read path: cold fill, resume, stream tuning.
//! GetObject read path: cold fill, resume, stream tuning.
use super::get_response::{GetObjectBody, GetObjectFields, GetObjectOutputFacts};
use super::head::ResponseEtag;
@@ -29,16 +29,12 @@ use crate::app::storage_api::object_usecase::object_cache::{
};
#[cfg(test)]
use crate::app::storage_api::s3::Body;
use crate::app::storage_api::s3::dto::{
Checksum, ChecksumType, GetObjectAttributesInput, GetObjectAttributesOutput, GetObjectAttributesParts, ObjectAttributes,
ObjectPart, Timestamp,
};
#[cfg(test)]
use crate::app::storage_api::s3::dto::{GetObjectInput, GetObjectOutput, StreamingBlob};
use crate::app::storage_api::s3::header::X_AMZ_RESTORE;
#[cfg(test)]
use crate::app::storage_api::s3::stream::ByteStream;
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Result, s3_error};
use crate::on_demand_migration::WriteBackBody;
struct ConditionalRead {
@@ -4950,221 +4946,6 @@ impl DefaultObjectUsecase {
record_get_object_s3_handler_stage_duration(GET_OBJECT_STAGE_RESPONSE_FINALIZE, response_finalize_start);
response
}
pub async fn execute_get_object_attributes(
&self,
req: S3Request<GetObjectAttributesInput>,
) -> S3Result<S3Response<GetObjectAttributesOutput>> {
if let Some(context) = &self.context {
let _ = context.object_store();
}
let mut helper =
OperationHelper::new(&req, EventName::ObjectAccessedAttributes, S3Operation::GetObjectAttributes).suppress_event();
let GetObjectAttributesInput {
bucket,
key,
max_parts,
object_attributes,
part_number_marker,
version_id,
sse_customer_key,
sse_customer_key_md5,
..
} = req.input;
let Some(store) = self.object_store() else {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let mut opts: ObjectOptions = get_opts(&bucket, &key, version_id.clone(), None, &req.headers)
.await
.map_err(ApiError::from)?;
opts.include_part_checksums = object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_PARTS);
let info = match store.get_object_info(&bucket, &key, &opts).await {
Ok(info) => info,
Err(err) => {
if is_err_object_not_found(&err) || is_err_version_not_found(&err) {
if is_dir_object(&key) {
let has_children = match probe_prefix_has_children(store, &bucket, &key, false).await {
Ok(has_children) => has_children,
Err(e) => {
error!(
"Failed to probe children for object attributes (bucket: {}, key: {}): {}",
bucket, key, e
);
false
}
};
let msg = head_prefix_not_found_message(&bucket, &key, has_children);
return Err(S3Error::with_message(S3ErrorCode::NoSuchKey, msg));
}
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
}
return Err(ApiError::from(err).into());
}
};
if info.delete_marker {
if opts.version_id.is_none() {
return Err(S3Error::new(S3ErrorCode::NoSuchKey));
}
return Err(S3Error::new(S3ErrorCode::MethodNotAllowed));
}
validate_ssec_for_read(&info.user_defined, sse_customer_key.as_ref(), sse_customer_key_md5.as_ref())?;
let metadata_map = info.user_defined.clone();
debug!(
"GetObjectAttributes raw object_attributes={:?}",
object_attributes.iter().map(|value| value.as_str()).collect::<Vec<_>>()
);
let requested = |name: &'static str| -> bool { object_attributes_requested(&object_attributes, name) };
let storage_class =
response_storage_class_for_object_attributes(&info, &metadata_map, requested(ObjectAttributes::STORAGE_CLASS));
let e_tag = if requested(ObjectAttributes::ETAG) {
info.etag.as_ref().map(|etag| to_s3s_etag(etag))
} else {
None
};
let object_size = if requested(ObjectAttributes::OBJECT_SIZE) {
Some(info.get_actual_size().map_err(ApiError::from)?)
} else {
None
};
let checksum = if requested(ObjectAttributes::CHECKSUM) {
let (checksums, is_multipart) = info.decrypt_checksums(0, &req.headers).map_err(ApiError::from)?;
// GetObjectAttributes returns checksums in the XML body, and s3s's Checksum
// type has no field for the additional algorithms, so `extra` cannot be
// surfaced here (unlike the header-based GET/HEAD paths) — an s3s limitation
// tracked for when it gains typed fields.
let ResponseChecksums {
crc32: checksum_crc32,
crc32c: checksum_crc32c,
sha1: checksum_sha1,
sha256: checksum_sha256,
crc64nvme: checksum_crc64nvme,
checksum_type,
..
} = classify_response_checksums(checksums, is_multipart);
Some(Checksum {
checksum_crc32,
checksum_crc32c,
checksum_sha1,
checksum_sha256,
checksum_crc64nvme,
checksum_type: checksum_type.map(ChecksumType::from),
..Default::default()
})
} else {
None
};
let object_parts = if requested(ObjectAttributes::OBJECT_PARTS) && info.is_multipart() {
let params = parse_list_parts_params(part_number_marker, max_parts)?;
let mut parts = Vec::new();
let mut marker = params.part_number_marker;
let max_parts = params.max_parts;
let mut start_at = 0usize;
if let Some(marker_value) = marker {
if let Some(index) = info.parts.iter().position(|part| part.number == marker_value) {
start_at = index + 1;
} else {
marker = None;
}
}
let max_parts: i32 = max_parts.try_into().map_err(|_| {
S3Error::with_message(S3ErrorCode::InvalidArgument, "max-parts value is out of range".to_string())
})?;
let end = (start_at + params.max_parts).min(info.parts.len());
let is_truncated = end < info.parts.len();
for part in &info.parts[start_at..end] {
let (checksums, is_multipart) = info.decrypt_checksums(part.number, &req.headers).map_err(ApiError::from)?;
// Additional algorithms cannot be surfaced in the ObjectPart XML body
// (s3s has no field); same limitation as the object-level attributes above.
let ResponseChecksums {
crc32: checksum_crc32,
crc32c: checksum_crc32c,
sha1: checksum_sha1,
sha256: checksum_sha256,
crc64nvme: checksum_crc64nvme,
..
} = classify_response_checksums(checksums, is_multipart);
let part_size = if part.actual_size > 0 {
part.actual_size
} else {
part.size.try_into().map_err(|_| {
S3Error::with_message(S3ErrorCode::InvalidArgument, "Part size value is out of range".to_string())
})?
};
parts.push(ObjectPart {
checksum_crc32,
checksum_crc32c,
checksum_sha1,
checksum_sha256,
checksum_crc64nvme,
part_number: i32::try_from(part.number).ok(),
size: Some(part_size),
..Default::default()
});
}
let part_number_marker = marker.and_then(|v| i32::try_from(v).ok());
let next_part_number_marker = parts.last().and_then(|part| part.part_number);
Some(GetObjectAttributesParts {
is_truncated: Some(is_truncated),
max_parts: Some(max_parts),
next_part_number_marker,
part_number_marker,
parts: Some(parts),
total_parts_count: Some(i32::try_from(info.parts.len()).map_err(|_| {
S3Error::with_message(S3ErrorCode::InvalidArgument, "Part count is out of range".to_string())
})?),
})
} else {
None
};
let version_id = s3_response_version_id(info.version_id);
let output = GetObjectAttributesOutput {
checksum,
delete_marker: if info.delete_marker { Some(true) } else { None },
e_tag,
last_modified: info.mod_time.map(Timestamp::from),
object_parts,
object_size,
storage_class,
version_id: version_id.clone(),
..Default::default()
};
helper = helper.object(info).version_id(version_id.unwrap_or_default());
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
result
}
}
fn object_attributes_requested(object_attributes: &[ObjectAttributes], name: &'static str) -> bool {
object_attributes.iter().any(|value| {
value.as_str().split(',').any(|part| {
part.trim_matches(|c: char| c.is_whitespace() || c == '"' || c == '\'')
.eq_ignore_ascii_case(name)
})
})
}
/// Bytes the inline tee may queue for the write-back ahead of the client.
@@ -12598,57 +12379,6 @@ mod tests {
let err = Box::pin(usecase.execute_get_object_from_s3s(req)).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
}
#[tokio::test]
async fn execute_get_object_attributes_returns_internal_error_when_store_uninitialized() {
let input = GetObjectAttributesInput::builder()
.bucket("test-bucket".to_string())
.key("test-key".to_string())
.build()
.unwrap();
let req = build_request(input, Method::GET);
let usecase = DefaultObjectUsecase::without_context();
let err = usecase.execute_get_object_attributes(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InternalError);
}
#[test]
fn object_attributes_requested_with_single_value() {
let object_attributes = vec![ObjectAttributes::from_static(ObjectAttributes::ETAG)];
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::ETAG));
assert!(!object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_SIZE));
}
#[test]
fn object_attributes_requested_with_comma_separated_values() {
let object_attributes = vec![
ObjectAttributes::from_static("ObjectParts,etag"),
ObjectAttributes::from_static("StorageClass"),
];
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_PARTS));
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::ETAG));
assert!(!object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_SIZE));
}
#[test]
fn object_attributes_requested_with_quotes_and_spaces() {
let object_attributes = vec![ObjectAttributes::from_static("'ObjectSize', \"Checksum\" , \"Etag\"")];
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_SIZE));
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::CHECKSUM));
assert!(object_attributes_requested(&object_attributes, ObjectAttributes::ETAG));
}
#[test]
fn object_attributes_requested_returns_false_for_missing_name() {
let object_attributes = vec![ObjectAttributes::from_static("Checksum")];
assert!(!object_attributes_requested(&object_attributes, ObjectAttributes::OBJECT_SIZE));
}
}
#[cfg(test)]
+26 -10
View File
@@ -86,8 +86,7 @@ use super::storage_api::object_usecase::error::{
};
use super::storage_api::object_usecase::head_prefix::{head_prefix_not_found_message, probe_prefix_has_children};
use super::storage_api::object_usecase::helper::{
OperationHelper, build_event_resp_elements, build_owned_event_resp_elements, run_owned_mutation,
spawn_background_with_context,
OperationHelper, build_owned_event_resp_elements, run_owned_mutation, spawn_background_with_context,
};
use super::storage_api::object_usecase::io::{
DynReader, EtagAlgorithm, HashReader, WritePlan, compression_metadata_value, wrap_reader,
@@ -97,6 +96,7 @@ use super::storage_api::object_usecase::object_cache::GetObjectBodySource;
#[cfg(test)]
use super::storage_api::object_usecase::object_cache::lookup_get_object_body_cache_hook;
use super::storage_api::object_usecase::object_cache::{GetObjectBodyCacheHookLookup, get_object_body_cache_plaintext_len};
#[cfg(test)]
use super::storage_api::object_usecase::object_utils::to_s3s_etag;
use super::storage_api::object_usecase::options::{
copy_dst_opts_with_replication_authorization, copy_src_opts, del_opts_with_versioning, extract_metadata,
@@ -144,6 +144,7 @@ use rustfs_audit::ObjectVersion as AuditObjectVersion;
use rustfs_concurrency::GetObjectQueueSnapshot;
use rustfs_config::MI_B;
use rustfs_filemeta::{NULL_VERSION_ID, RestoreStatusOps, parse_restore_obj_status};
use rustfs_gateway_types::dto::SelectObjectContentOutput;
use rustfs_gateway_types::persistence::PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration;
use rustfs_io_core::{BytesPool, PooledBuffer};
use rustfs_io_metrics;
@@ -151,6 +152,7 @@ use rustfs_notify::EventArgsBuilder;
use rustfs_object_capacity::capacity_manager::get_capacity_manager;
use rustfs_policy::policy::action::{Action, S3Action};
use rustfs_s3_ops::{S3Operation, delete_event_name_for_marker, put_event_name_for_post_object};
use rustfs_s3select_api::request::SelectObjectContentInput;
use rustfs_targets::{EventName, get_request_host, get_request_port, get_request_user_agent};
use rustfs_utils::CompressionAlgorithm;
#[cfg(test)]
@@ -177,8 +179,9 @@ use rustfs_utils::http::{
use rustfs_utils::path::{encode_dir_object, is_dir_object, path_join_buf};
use rustfs_utils::retry::{DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, RetryTimer};
use rustfs_zip::{ArchiveLimits, CompressionFormat, ZipError};
use s3s::dto::{SelectObjectContentInput, SelectObjectContentOutput};
use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use s3s::{S3Error, S3ErrorCode, S3Result};
#[cfg(test)]
use s3s::{S3Request, S3Response};
fn object_s3_error(code: S3ErrorCode, message: impl Into<std::borrow::Cow<'static, str>>) -> S3Error {
S3Error::with_message(code, message)
@@ -204,12 +207,20 @@ fn remove_source_replication_bookkeeping(user_defined: &mut HashMap<String, Stri
}
}
mod attributes;
#[cfg(test)]
mod attributes_completion_test;
#[cfg(test)]
mod conditional_read_bench;
pub(crate) use attributes::{GetObjectAttributesFields, GetObjectAttributesOutputFacts};
mod copy;
mod copy_input;
pub(crate) use copy_input::{CopyObjectWriteInput, CopyObjectWriteOutput, CopySourceCondition};
#[cfg(all(any(feature = "cuobject", feature = "nixl"), target_os = "linux"))]
pub(crate) mod cuobject;
mod delete;
mod delete_batch_types;
pub(crate) use delete_batch_types::*;
mod extract;
mod get;
mod get_response;
@@ -375,16 +386,21 @@ impl DefaultObjectUsecase {
map_quota_check_outcome(bucket, quota_checker.check_quota(bucket, op, size).await).map(Some)
}
#[hotpath::measure(impl_type = "DefaultObjectUsecase", future = true)]
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_select_object_content(
#[instrument(name = "execute_select_object_content", level = "debug", skip(self, envelope, input))]
#[hotpath::measure(
impl_type = "DefaultObjectUsecase",
future = true,
label = "rustfs::app::object::DefaultObjectUsecase::execute_select_object_content"
)]
pub(crate) async fn execute_select_object_content_owned(
&self,
req: S3Request<SelectObjectContentInput>,
) -> S3Result<S3Response<SelectObjectContentOutput>> {
envelope: crate::app::RequestEnvelope,
input: SelectObjectContentInput,
) -> rustfs_s3_types::S3Result<crate::app::Answered<SelectObjectContentOutput>> {
if let Some(context) = &self.context {
let _ = context.object_store();
}
crate::app::select_object::execute_select_object_content(req).await
crate::app::select_object::execute_select_object_content(envelope, input).await
}
}
+46 -43
View File
@@ -15,11 +15,12 @@
//! RestoreObject path.
use super::*;
use crate::app::storage_api::object_usecase::bucket::restore_request_from_wire;
use crate::app::storage_api::s3::dto::{RequestCharged, RestoreObjectInput, RestoreObjectOutput, RestoreRequestType};
use crate::app::storage_api::s3::header::X_AMZ_RESTORE;
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use crate::app::RequestEnvelope;
use crate::app::storage_api::object_usecase::bucket::RestoreRequest;
use rustfs_filemeta::RestoreStatus;
use rustfs_gateway_types::dto::{RequestCharged, RestoreObjectOutput};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
use rustfs_utils::http::headers::AMZ_RESTORE;
// RUSTFS_COMPAT_TODO(backlog-1337): legacy restores lack a liveness marker. Remove after the minimum supported release writes v1 on every restore.
const LEGACY_RESTORE_ORPHAN_GRACE: time::Duration = time::Duration::hours(24);
@@ -153,20 +154,24 @@ async fn maybe_pause_after_restore_status_commit(bucket: &str, object: &str) {
}
impl DefaultObjectUsecase {
#[instrument(level = "debug", skip(self, req))]
pub async fn execute_restore_object(&self, req: S3Request<RestoreObjectInput>) -> S3Result<S3Response<RestoreObjectOutput>> {
#[instrument(
name = "execute_restore_object",
level = "debug",
skip(self, envelope, bucket, object, version_id, restore_request)
)]
pub(crate) async fn execute_restore_object_owned(
&self,
envelope: RequestEnvelope,
bucket: String,
object: String,
version_id: Option<String>,
restore_request: Option<RestoreRequest>,
) -> S3Result<RestoreObjectOutput> {
if let Some(context) = &self.context {
let _ = context.object_store();
}
let mut helper = OperationHelper::new(&req, EventName::ObjectRestorePost, S3Operation::RestoreObject);
let RestoreObjectInput {
bucket,
key: object,
restore_request: rreq,
version_id,
..
} = req.input.clone();
let helper = OperationHelper::from_view(&envelope.audit_view(), EventName::ObjectRestorePost, S3Operation::RestoreObject);
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &object).await?;
validate_table_catalog_object_mutation(&bucket, &object).await?;
@@ -174,7 +179,8 @@ impl DefaultObjectUsecase {
// Typed S3 errors on every RestoreObject failure (backlog#2205): a
// `Custom` code serializes as a generic 500, which makes SDK clients
// retry client errors and conflicts alike.
let rreq = rreq.ok_or_else(|| S3Error::with_message(S3ErrorCode::MalformedXML, "restore request is required"))?;
let rreq =
restore_request.ok_or_else(|| S3Error::with_message(S3ErrorCode::MalformedXML, "restore request is required"))?;
// SELECT-type restore is not supported (backlog#1341). The restore
// path can only write the retrieved bytes back to the source key, so
@@ -184,11 +190,7 @@ impl DefaultObjectUsecase {
// on a versioned one) while never writing anything to
// `OutputLocation.S3`. Reject before any guard, metadata write or
// fabricated `x-amz-restore-output-path` response header.
if rreq
.type_
.as_ref()
.is_some_and(|type_| type_.as_str() == RestoreRequestType::SELECT)
{
if rreq.type_.as_deref() == Some("SELECT") {
return Err(S3Error::with_message(
S3ErrorCode::NotImplemented,
"SELECT restore requests are not supported.",
@@ -199,8 +201,6 @@ impl DefaultObjectUsecase {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let rreq = restore_request_from_wire(rreq);
// Validate the request shape before taking any lock or reading the
// object: a malformed request or an illegal `Days` value is a client
// error, and the validator messages are static — they carry no
@@ -212,11 +212,11 @@ impl DefaultObjectUsecase {
));
}
let version_id_str = version_id.clone().unwrap_or_default();
let version_id_str = version_id.unwrap_or_default();
let mut opts = post_restore_opts(&version_id_str, &bucket, &object)
.await
.map_err(ApiError::from)?;
apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
apply_bucket_generation_guard(&envelope, &bucket, &mut opts)?;
// `apply_bucket_generation_guard` deliberately tolerates a missing guard
// (only the S3 access layer installs one), so this must not hard-require
// it. Resolve the current generation instead, exactly as the copy path
@@ -370,7 +370,7 @@ impl DefaultObjectUsecase {
metadata.insert(AMZ_RESTORE_REQUEST_DATE.to_string(), request_date);
if already_restored {
metadata.insert(
X_AMZ_RESTORE.as_str().to_string(),
AMZ_RESTORE.to_owned(),
RestoreStatus {
is_restore_in_progress: Some(false),
restore_expiry_date: Some(restore_expiry),
@@ -379,7 +379,7 @@ impl DefaultObjectUsecase {
);
} else {
metadata.insert(
X_AMZ_RESTORE.as_str().to_string(),
AMZ_RESTORE.to_owned(),
RestoreStatus {
is_restore_in_progress: Some(true),
restore_expiry_date: Some(OffsetDateTime::now_utc()),
@@ -444,16 +444,12 @@ impl DefaultObjectUsecase {
drop(superseded_worker_guard.take());
if already_restored {
let output = RestoreObjectOutput {
request_charged: Some(RequestCharged::from_static(RequestCharged::REQUESTER)),
restore_output_path: None,
};
helper = helper
let result = Ok(restore_output());
let _ = helper
.object(event_object_info.clone())
.version_id(version_id_str.clone())
.suppress_event();
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
.suppress_event()
.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
return result;
}
}
@@ -485,7 +481,7 @@ impl DefaultObjectUsecase {
}
let restore_worker_guard = restore_worker_guard.take();
let table_data_plane_publication_guards = retained_table_data_plane_publication_guards(&req);
let table_data_plane_publication_guards = retained_table_data_plane_publication_guards(&envelope);
spawn_traced(async move {
let _restore_worker_guard = restore_worker_guard;
let _table_data_plane_publication_guards = table_data_plane_publication_guards;
@@ -526,22 +522,29 @@ impl DefaultObjectUsecase {
}
});
let output = RestoreObjectOutput {
request_charged: Some(RequestCharged::from_static(RequestCharged::REQUESTER)),
restore_output_path: None,
};
helper = helper.object(event_object_info).version_id(version_id_str);
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
let result = Ok(restore_output());
let _ = helper.object(event_object_info).version_id(version_id_str).complete_owned(
&result,
StatusCode::OK,
&HeaderMap::new(),
);
result
}
}
fn restore_output() -> RestoreObjectOutput {
RestoreObjectOutput {
request_charged: Some(RequestCharged::custom("requester".to_owned())),
restore_output_path: None,
}
}
#[cfg(test)]
mod tests {
use super::*;
use http::Method;
use s3s::dto::RestoreRequest;
use s3s::S3ErrorCode;
use s3s::dto::{RestoreObjectInput, RestoreRequest};
fn ongoing_metadata(operation_id: Uuid) -> HashMap<String, String> {
let mut metadata = HashMap::new();
+9 -14
View File
@@ -17,9 +17,9 @@
use super::*;
#[cfg(test)]
use crate::app::storage_api::s3::dto::ObjectLockRetentionMode;
use crate::app::storage_api::s3::dto::{
ChecksumAlgorithm, ObjectLockLegalHoldStatus, ObjectLockMode, ServerSideEncryption, StorageClass, Timestamp, TimestampFormat,
};
use crate::app::storage_api::s3::dto::{ChecksumAlgorithm, ServerSideEncryption, Timestamp, TimestampFormat};
#[cfg(test)]
use crate::app::storage_api::s3::dto::{ObjectLockLegalHoldStatus, ObjectLockMode};
use crate::app::storage_api::s3::header::{X_AMZ_DELETE_MARKER, X_AMZ_VERSION_ID};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Result, StdError, s3_error};
use crate::on_demand_migration::{OdmStateError, PolicyConfig, SourceErrorPolicy, SourceHead};
@@ -815,7 +815,7 @@ pub(super) fn response_storage_class_for_object_attributes(
info: &ObjectInfo,
metadata: &HashMap<String, String>,
requested: bool,
) -> Option<StorageClass> {
) -> Option<String> {
if !requested {
return None;
}
@@ -828,9 +828,7 @@ pub(super) fn response_storage_class_for_object_attributes(
&& !info.transitioned_object.tier.is_empty())
.then_some(info.transitioned_object.tier.as_str());
Some(StorageClass::from(
storageclass::effective_class(stored_class, transitioned_tier).to_string(),
))
Some(storageclass::effective_class(stored_class, transitioned_tier).to_string())
}
// Shared across Object Lock validation paths to keep the client-facing
@@ -838,7 +836,8 @@ pub(super) fn response_storage_class_for_object_attributes(
pub(crate) const ERR_OBJECT_LOCK_RETENTION_HEADERS_MUST_BE_PAIRED: &str =
"x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied";
pub(crate) fn build_put_like_object_lock_metadata(
#[cfg(test)]
fn build_put_like_object_lock_metadata(
bucket: &str,
object_lock_config_state: &metadata_sys::ObjectLockConfigState,
object_lock_legal_hold_status: Option<ObjectLockLegalHoldStatus>,
@@ -1952,9 +1951,7 @@ mod tests {
};
assert_eq!(
response_storage_class_for_object_attributes(&info, &metadata, true)
.as_ref()
.map(StorageClass::as_str),
response_storage_class_for_object_attributes(&info, &metadata, true).as_deref(),
Some(storageclass::STANDARD)
);
@@ -1963,9 +1960,7 @@ mod tests {
..Default::default()
};
assert_eq!(
response_storage_class_for_object_attributes(&legacy_info, &HashMap::new(), true)
.as_ref()
.map(StorageClass::as_str),
response_storage_class_for_object_attributes(&legacy_info, &HashMap::new(), true).as_deref(),
Some(storageclass::STANDARD)
);
}
+119 -2
View File
@@ -16,23 +16,140 @@
//!
//! Responsible for: reading the stored status, `OFF` when none is stored, on a bucket with Object
//! Lock enabled. A stored status outside the model's set is answered as stored.
//! NOT responsible for: authorization, or setting a hold (PutObjectLegalHold).
//! Also writes lock metadata to the requested version and schedules metadata replication.
//! NOT responsible for: authorization.
//! Upstream: both edges. Downstream: the object store and the bucket's Object Lock configuration.
use super::{ObjectMetaUsecase, locked_object_info};
use crate::app::RequestEnvelope;
use crate::app::object::shared::{reject_veeam_sosapi_mutation, validate_table_catalog_object_mutation};
use crate::app::storage_api::object_meta_usecase::contract::ObjectOperations as _;
use crate::app::storage_api::object_meta_usecase::contract::{BucketOperations as _, BucketOptions};
use crate::app::storage_api::object_meta_usecase::validate_bucket_object_lock_enabled;
use crate::app::storage_api::object_usecase::access::{RequestedObjectLegalHold, apply_bucket_generation_guard};
use crate::app::storage_api::object_usecase::bucket::replication::{
ReplicateDecision, must_replicate_metadata, schedule_metadata_replication,
};
use crate::app::storage_api::object_usecase::helper::OperationHelper;
use crate::app::storage_api::object_usecase::options::get_opts;
use crate::app::storage_api::object_usecase::{StorageObjectOptions as ObjectOptions, parse_object_lock_legal_hold_native};
use crate::error::ApiError;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::GetObjectLegalHoldInput;
use rustfs_gateway_types::dto::{GetObjectLegalHoldOutput, ObjectLockLegalHold, Status};
use rustfs_gateway_types::dto::{PutObjectLegalHoldOutput, RequestCharged};
use rustfs_s3_ops::S3Operation;
use rustfs_s3_types::{EventName, S3Result};
use rustfs_s3_types::{EventName, S3Error, S3ErrorCode, S3Result};
use rustfs_utils::http::headers::AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER;
use rustfs_utils::http::{SUFFIX_REPLICATION_GENERATION, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, insert_str};
use tracing::error;
use uuid::Uuid;
impl ObjectMetaUsecase {
/// Updates the lock metadata of the version named by the request.
pub(crate) async fn put_object_legal_hold_fields(
&self,
envelope: RequestEnvelope,
bucket: &str,
key: &str,
version_id: Option<String>,
legal_hold: Option<RequestedObjectLegalHold>,
) -> S3Result<PutObjectLegalHoldOutput> {
let helper = OperationHelper::from_view(
&envelope.audit_view(),
EventName::ObjectCreatedPutLegalHold,
S3Operation::PutObjectLegalHold,
)
.suppress_event();
reject_veeam_sosapi_mutation(|| self.object_store().ok(), bucket, key).await?;
validate_table_catalog_object_mutation(bucket, key).await?;
let store = self.object_store()?;
if let Err(err) = validate_bucket_object_lock_enabled(bucket).await {
let _ = store
.get_bucket_info(bucket, &BucketOptions::default())
.await
.map_err(ApiError::from)?;
return Err(err.into());
}
let opts: ObjectOptions = get_opts(bucket, key, version_id.clone(), None, envelope.headers())
.await
.map_err(ApiError::from)?;
let mut popts = ObjectOptions {
mod_time: opts.mod_time,
version_id: opts.version_id.clone(),
..Default::default()
};
apply_bucket_generation_guard(&envelope, bucket, &mut popts)?;
// PutObjectLegalHold only rewrites metadata, so replication is not scheduled by the
// object PUT path. Schedule it explicitly, otherwise the legal hold never reaches the
// replica and the peer copy remains deletable.
//
// Mirror the PUT path ordering (see object_usecase::put_object): compute the decision
// once BEFORE the commit and persist the pending marker with it, so a restart between
// the commit and the worker write-back leaves a Pending status on disk for the scanner
// to re-drive. Scheduling without that marker would lose the task silently.
let dsc = if crate::app::storage_api::object_usecase::bucket::metadata_sys::get_replication_config(bucket)
.await
.is_ok()
&& let Ok(info) = store.get_object_info(bucket, key, &opts).await
{
must_replicate_metadata(
bucket,
key,
&info.user_defined,
info.user_tags.as_ref().clone(),
popts.delete_marker_replication_status(),
popts.clone(),
)
.await
} else {
ReplicateDecision::new()
};
let mut eval_metadata = parse_object_lock_legal_hold_native(legal_hold)?;
if dsc.replicate_any() {
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_GENERATION, Uuid::new_v4().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, dsc.pending_status().unwrap_or_default());
}
popts.eval_metadata = Some(eval_metadata);
let info = store.put_object_metadata(bucket, key, &popts).await.map_err(|e| {
error!(
event = "object_metadata_write_failed",
component = "storage",
subsystem = "object",
error = %e,
"Failed to write object metadata"
);
S3Error::with_message(S3ErrorCode::InternalError, e.to_string())
})?;
// The current target transport carries the updated metadata in a full-object PUT.
if dsc.replicate_any() {
schedule_metadata_replication(info.clone(), store.clone(), dsc).await;
}
let output = PutObjectLegalHoldOutput {
request_charged: Some(RequestCharged::REQUESTER),
};
let version_id = version_id.unwrap_or_default();
let helper = helper.object(info).version_id(version_id);
let result = Ok(output);
let _ = helper.complete_owned(&result, http::StatusCode::OK, &http::HeaderMap::new());
rustfs_scanner::record_dirty_usage_object_from_producer(
bucket,
key,
rustfs_scanner::SegmentInvalidationProducerIdentity::ObjectMetadata,
);
result
}
/// The legal hold of the object `input` names.
///
/// # Errors
+137 -1
View File
@@ -17,24 +17,160 @@
//! Responsible for: reading the stored mode and retain-until date on a bucket with Object Lock
//! enabled. A stored mode outside the model's set is answered as stored; a stored date is answered
//! at its instant in UTC, and an instant outside the years 0 to 9999 in UTC is `InternalError`.
//! NOT responsible for: authorization, or setting a retention (PutObjectRetention).
//! Also writes lock metadata to the requested version and schedules metadata replication.
//! NOT responsible for: authorization.
//! Upstream: both edges. Downstream: the object store and the bucket's Object Lock configuration.
use super::{ObjectMetaUsecase, locked_object_info};
use crate::app::RequestEnvelope;
use crate::app::dto_values;
use crate::app::object::shared::{reject_veeam_sosapi_mutation, validate_table_catalog_object_mutation};
use crate::app::storage_api::object_meta_usecase::contract::{ObjectLockRetentionOptions, ObjectOperations as _};
use crate::app::storage_api::object_meta_usecase::{check_retention_for_modification, validate_bucket_object_lock_enabled};
use crate::app::storage_api::object_usecase::access::{
RequestedObjectLockRetention, apply_bucket_generation_guard, has_bypass_governance_header,
};
use crate::app::storage_api::object_usecase::bucket::replication::{
ReplicateDecision, must_replicate_metadata, schedule_metadata_replication,
};
use crate::app::storage_api::object_usecase::helper::OperationHelper;
use crate::app::storage_api::object_usecase::options::get_opts;
use crate::app::storage_api::object_usecase::{StorageObjectOptions as ObjectOptions, parse_object_lock_retention_native};
use crate::error::ApiError;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::GetObjectRetentionInput;
use rustfs_gateway_types::dto::{GetObjectRetentionOutput, Mode, ObjectLockRetention};
use rustfs_gateway_types::dto::{PutObjectRetentionOutput, RequestCharged};
use rustfs_s3_ops::S3Operation;
use rustfs_s3_types::{EventName, S3Error, S3ErrorCode, S3Result};
use rustfs_utils::http::{AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER};
use rustfs_utils::http::{SUFFIX_REPLICATION_GENERATION, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TIMESTAMP, insert_str};
use std::collections::HashMap;
use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
use tracing::error;
use uuid::Uuid;
impl ObjectMetaUsecase {
/// Updates the lock metadata of the version named by the request.
pub(crate) async fn put_object_retention_fields(
&self,
envelope: RequestEnvelope,
bucket: &str,
key: &str,
version_id: Option<String>,
retention: Option<RequestedObjectLockRetention>,
) -> S3Result<PutObjectRetentionOutput> {
let helper = OperationHelper::from_view(
&envelope.audit_view(),
EventName::ObjectCreatedPutRetention,
S3Operation::PutObjectRetention,
)
.suppress_event();
reject_veeam_sosapi_mutation(|| self.object_store().ok(), bucket, key).await?;
validate_table_catalog_object_mutation(bucket, key).await?;
let store = self.object_store()?;
validate_bucket_object_lock_enabled(bucket).await?;
let new_retain_until = retention.as_ref().and_then(|retention| retention.retain_until);
let new_mode = retention.as_ref().and_then(|retention| retention.mode.clone());
let bypass_governance = has_bypass_governance_header(envelope.headers());
// Keep the early check for existing response behavior; put_object_metadata
// repeats the same check after taking the metadata write lock.
let check_opts: ObjectOptions = get_opts(bucket, key, version_id.clone(), None, envelope.headers())
.await
.map_err(ApiError::from)?;
let existing_obj_info = store.get_object_info(bucket, key, &check_opts).await.ok();
if let Some(existing_obj_info) = existing_obj_info.as_ref()
&& let Some(block_reason) = check_retention_for_modification(
&existing_obj_info.user_defined,
new_mode.as_deref(),
new_retain_until,
bypass_governance,
)
{
return Err(S3Error::with_message(S3ErrorCode::AccessDenied, block_reason.error_message()));
}
let mut opts: ObjectOptions = get_opts(bucket, key, version_id.clone(), None, envelope.headers())
.await
.map_err(ApiError::from)?;
apply_bucket_generation_guard(&envelope, bucket, &mut opts)?;
opts.object_lock_retention = Some(ObjectLockRetentionOptions {
mode: new_mode,
retain_until: new_retain_until,
bypass_governance,
});
// PutObjectRetention only rewrites metadata, so the object PUT path that normally
// computes a replication decision and schedules replication never runs for it. Without
// scheduling here the peer keeps the previous, unprotected lock state and a
// WORM-protected object stays deletable on the replica.
//
// Mirror the PUT path ordering (see object_usecase::put_object): compute the decision
// once BEFORE the commit and persist the pending marker with it, so a restart between
// the commit and the worker write-back leaves a Pending status on disk for the scanner
// to re-drive. Scheduling without that marker would lose the task silently.
let dsc = match existing_obj_info.as_ref() {
Some(info) => {
must_replicate_metadata(
bucket,
key,
&info.user_defined,
info.user_tags.as_ref().clone(),
opts.delete_marker_replication_status(),
opts.clone(),
)
.await
}
None => ReplicateDecision::new(),
};
let mut eval_metadata = parse_object_lock_retention_native(retention)?;
if dsc.replicate_any() {
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_GENERATION, Uuid::new_v4().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, dsc.pending_status().unwrap_or_default());
}
opts.eval_metadata = Some(eval_metadata);
let object_info = store.put_object_metadata(bucket, key, &opts).await.map_err(|e| {
error!(
event = "object_metadata_write_failed",
component = "storage",
subsystem = "object",
error = %e,
"Failed to write object metadata"
);
S3Error::from(ApiError::from(e))
})?;
// The current target transport carries the updated metadata in a full-object PUT.
if dsc.replicate_any() {
schedule_metadata_replication(object_info.clone(), store.clone(), dsc).await;
}
let output = PutObjectRetentionOutput {
request_charged: Some(RequestCharged::REQUESTER),
};
let version_id = version_id.unwrap_or_else(|| Uuid::new_v4().to_string());
let helper = helper.object(object_info).version_id(version_id);
let result = Ok(output);
let _ = helper.complete_owned(&result, http::StatusCode::OK, &http::HeaderMap::new());
rustfs_scanner::record_dirty_usage_object_from_producer(
bucket,
key,
rustfs_scanner::SegmentInvalidationProducerIdentity::ObjectMetadata,
);
result
}
/// The retention of the object `input` names.
///
/// # Errors
+152 -9
View File
@@ -12,20 +12,21 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! GetObjectTagging and DeleteObjectTagging: the tag set an object version stores.
//! GetObjectTagging, PutObjectTagging and DeleteObjectTagging for an object version.
//!
//! Responsible for: reading the stored tags (as stored), deleting them with metadata replication,
//! and, for an object missing locally, serving the request from a replication target that has it.
//! NOT responsible for: authorization (and so not the version swap of its action), or
//! PutObjectTagging, which still runs on the legacy edge.
//! Reads stored tags and applies validated mutations with metadata replication. For an object
//! missing locally, it can serve the request from a replication target that has it. The edges
//! authorize requests before invoking these operations.
//! Upstream: both edges. Downstream: the object store, metadata replication and the tagging proxy.
use super::ObjectMetaUsecase;
use super::tagging_proxy::{proxy_delete_object_tagging, proxy_get_object_tagging};
use super::tagging_proxy::{proxy_delete_object_tagging, proxy_get_object_tagging, proxy_put_object_tagging};
use crate::app::RequestEnvelope;
use crate::app::object::shared::{reject_veeam_sosapi_mutation, s3_response_version_id, validate_table_catalog_object_mutation};
use crate::app::storage_api::object_meta_usecase::contract::ObjectOperations as _;
use crate::app::storage_api::object_meta_usecase::{VersioningApi as _, decode_stored_tags, parse_object_version_id};
use crate::app::storage_api::object_meta_usecase::{
VersioningApi as _, decode_stored_tags, encode_stored_tags, parse_object_version_id, validate_tag_set_entries,
};
use crate::app::storage_api::object_usecase::ECStore;
use crate::app::storage_api::object_usecase::StorageObjectOptions as ObjectOptions;
use crate::app::storage_api::object_usecase::bucket::replication::{must_replicate_metadata, schedule_metadata_replication};
@@ -36,9 +37,12 @@ use crate::app::storage_api::object_usecase::options::get_opts;
use crate::error::ApiError;
use http::{HeaderMap, StatusCode};
use metrics::{counter, histogram};
use rustfs_gateway_types::cors_tagging::PersistedTag;
#[cfg(feature = "gateway-stack")]
use rustfs_gateway_types::dto::PutObjectTaggingInput;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::{DeleteObjectTaggingInput, GetObjectTaggingInput};
use rustfs_gateway_types::dto::{DeleteObjectTaggingOutput, GetObjectTaggingOutput, Tag};
use rustfs_gateway_types::dto::{DeleteObjectTaggingOutput, GetObjectTaggingOutput, PutObjectTaggingOutput, Tag};
use rustfs_s3_ops::S3Operation;
use rustfs_s3_types::{EventName, S3Error, S3ErrorCode, S3Result};
use rustfs_utils::http::{
@@ -160,6 +164,130 @@ impl ObjectMetaUsecase {
})
}
/// Replaces the tag set of the named object version and replicates the change when configured.
///
/// # Errors
///
/// Returns `InvalidTag` for an invalid tag set, a refusal for reserved objects, or the
/// storage error when the tags cannot be written.
#[cfg(feature = "gateway-stack")]
pub async fn execute_put_object_tagging(
&self,
envelope: RequestEnvelope,
input: PutObjectTaggingInput,
) -> S3Result<PutObjectTaggingOutput> {
self.put_object_tagging_fields(
envelope,
input.bucket.as_str(),
input.key.as_str(),
input.version_id,
input.tagging.tag_set,
)
.await
}
pub(crate) async fn put_object_tagging_fields(
&self,
envelope: RequestEnvelope,
bucket: &str,
object: &str,
version_id: Option<String>,
tag_set: Vec<Tag>,
) -> S3Result<PutObjectTaggingOutput> {
let start_time = Instant::now();
let helper =
OperationHelper::from_view(&envelope.audit_view(), EventName::ObjectTaggingPut, S3Operation::PutObjectTagging);
let (bucket, object) = (bucket.to_owned(), object.to_owned());
reject_veeam_sosapi_mutation(|| self.object_store().ok(), &bucket, &object).await?;
validate_table_catalog_object_mutation(&bucket, &object).await?;
let (bucket, object) = (bucket.as_str(), object.as_str());
let persisted_tags = validate_put_tags(&tag_set)?;
let store = self.object_store()?;
let tags = encode_stored_tags(persisted_tags);
let mut opts = get_opts(bucket, object, version_id.clone(), None, envelope.headers())
.await
.map_err(ApiError::from)?;
let existing = match store.get_object_info(bucket, object, &opts).await {
Ok(info) => info,
Err(error) => {
// A write on a remote target emits no local object event.
if (is_err_object_not_found(&error) || is_err_version_not_found(&error))
&& proxy_allowed(&envelope)
&& proxy_put_object_tagging(bucket, object, version_id.clone(), envelope.headers(), &tag_set)
.await
.is_some()
{
counter!("rustfs_put_object_tagging_success").increment(1);
record_tagging_duration("put", start_time);
return Ok(PutObjectTaggingOutput { version_id });
}
return Err(ApiError::from(error).into());
}
};
let decision = must_replicate_metadata(
bucket,
object,
&existing.user_defined,
tags.clone(),
existing.replication_status,
opts.clone(),
)
.await;
if decision.replicate_any() {
let mut eval_metadata = HashMap::new();
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_GENERATION, Uuid::new_v4().to_string());
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
insert_str(
&mut eval_metadata,
SUFFIX_REPLICATION_STATUS,
decision.pending_status().unwrap_or_default(),
);
insert_str(
&mut eval_metadata,
SUFFIX_TAGGING_TIMESTAMP,
OffsetDateTime::now_utc().format(&Rfc3339).unwrap_or_default(),
);
opts.eval_metadata = Some(eval_metadata);
}
let info = store.put_object_tags(bucket, object, &tags, &opts).await.map_err(|error| {
error!(
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_TAGGING,
event = "object_tagging_put_failed",
bucket = %bucket,
object = %object,
error = %error,
"Failed to put object tags"
);
counter!("rustfs_put_object_tagging_failure").increment(1);
ApiError::from(error)
})?;
if decision.replicate_any() {
schedule_metadata_replication(info.clone(), Arc::clone(&store), decision).await;
}
counter!("rustfs_put_object_tagging_success").increment(1);
let event_version_id = version_id
.as_deref()
.filter(|value| !value.is_empty())
.map(str::to_owned)
.or_else(|| info.version_id.map(|version| version.to_string()))
.unwrap_or_default();
let result = Ok(PutObjectTaggingOutput { version_id });
let _ = helper
.object(info)
.version_id(event_version_id)
.complete_owned(&result, StatusCode::OK, &HeaderMap::new());
rustfs_scanner::record_dirty_usage_object_from_producer(
bucket,
object,
rustfs_scanner::SegmentInvalidationProducerIdentity::ObjectMetadata,
);
record_tagging_duration("put", start_time);
result
}
/// Deletes the tag set of the object version `input` names, replicating the change where the
/// bucket's rules ask for it. An object missing locally is deleted on a replication target that
/// has it, without a local event.
@@ -294,7 +422,7 @@ impl ObjectMetaUsecase {
/// A proxy may answer only when no tag-conditioned authorization ran: the remote's tags were never
/// shown to the policy that admitted the request.
fn proxy_allowed(envelope: &RequestEnvelope) -> bool {
pub(crate) fn proxy_allowed(envelope: &RequestEnvelope) -> bool {
envelope.object_tag_conditions().is_none()
}
@@ -313,3 +441,18 @@ fn stored_tags(encoded: &str) -> Vec<Tag> {
})
.collect()
}
fn validate_put_tags(tags: &[Tag]) -> S3Result<Vec<PersistedTag>> {
if tags.len() > 10 {
return Err(S3Error::with_message(S3ErrorCode::InvalidTag, "Cannot have more than 10 tags per object"));
}
let tags: Vec<_> = tags
.iter()
.map(|tag| PersistedTag {
key: tag.key.clone(),
value: tag.value.clone(),
})
.collect();
validate_tag_set_entries(&tags).map_err(|error| S3Error::with_message(S3ErrorCode::InvalidTag, error))?;
Ok(tags)
}
+1 -1
View File
@@ -19,7 +19,7 @@
//! target through the AWS SDK until one has the object, and the proxy metrics.
//! NOT responsible for: deciding whether proxying is allowed for the request (the caller checks
//! that no tag-conditioned authorization ran), or any local tagging.
//! Upstream: the tagging use cases, and the legacy PutObjectTagging edge until it moves here.
//! Upstream: the tagging use cases.
//! Downstream: the replication target clients.
use crate::app::storage_api::object_usecase::StorageObjectOptions as ObjectOptions;
+223 -204
View File
@@ -4,11 +4,11 @@ use super::storage_api::select_object::options::get_opts;
use super::storage_api::select_object::request_context::spawn_traced;
use super::storage_api::select_object::sse::{SseKmsPrincipal, authorize_sse_kms_object_read, project_sse_read_response_headers};
use super::storage_api::select_object::{
StoragePrepareSelectObjectSnapshotError, StorageSelectObjectSnapshot, get_validated_store, validate_sse_headers_for_read,
validate_ssec_for_read,
StoragePrepareSelectObjectSnapshotError, StorageSelectObjectSnapshot, get_validated_store_owned,
validate_sse_headers_for_read, validate_ssec_for_read,
};
use crate::app::runtime_sources::current_s3select_db;
use crate::app::select_request::select_input_from_s3s;
use crate::app::{Answered, RequestEnvelope};
use crate::error::ApiError;
use bytes::{Bytes, BytesMut};
use datafusion::arrow::{
@@ -27,19 +27,22 @@ use datafusion::common::DataFusionError;
use datafusion::physical_plan::SendableRecordBatchStream;
use futures::StreamExt;
use http::{HeaderMap, StatusCode, header::RANGE};
use rustfs_gateway_types::dto::{
ContinuationEvent, EndEvent, Progress, ProgressEvent, RecordsEvent,
SelectObjectContentEventStream as SelectObjectContentEvent, SelectObjectContentOutput, Stats, StatsEvent,
};
use rustfs_gateway_types::{SelectEventError, SelectEventStream};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
use rustfs_s3select_api::request::{
CompressionType, CsvOutput, ExpressionType, FileHeaderInfo, InputSerialization, JsonInput, JsonOutput, JsonType,
OutputSerialization, QuoteFields, SelectObjectContentInput, SelectObjectContentRequest,
};
use rustfs_s3select_api::{
QueryError, SelectError, SelectInputMetrics,
object_store::{INVALID_SCAN_RANGE_MESSAGE, is_noop_scan_range, validate_scan_range_bounds},
query::{Context, Query},
};
use rustfs_s3select_query::instance::s3_select_query_timeout;
use s3s::dto::{
CSVOutput, CompressionType, ContinuationEvent, EndEvent, ExpressionType, FileHeaderInfo, InputSerialization, JSONInput,
JSONOutput, JSONType, OutputSerialization, Progress, ProgressEvent, QuoteFields, RecordsEvent, SelectObjectContentEvent,
SelectObjectContentEventStream, SelectObjectContentInput, SelectObjectContentOutput, SelectObjectContentRequest, Stats,
StatsEvent,
};
use s3s::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use std::{
fmt,
future::poll_fn,
@@ -86,8 +89,8 @@ struct SelectValidation {
#[derive(Clone, Debug)]
enum SelectOutputFormat {
Csv(CSVOutput),
Json(JSONOutput),
Csv(CsvOutput),
Json(JsonOutput),
}
enum SelectProducerOutcome {
@@ -116,27 +119,24 @@ impl SelectSnapshotFence for Arc<StorageSelectObjectSnapshot> {
}
}
pub async fn execute_select_object_content(
req: S3Request<SelectObjectContentInput>,
) -> S3Result<S3Response<SelectObjectContentOutput>> {
let read_principal = SseKmsPrincipal::from_request(&req);
let mut input = req.input;
let validation = validate_select_request(&req.headers, &mut input)?;
pub(super) async fn execute_select_object_content(
envelope: RequestEnvelope,
mut input: SelectObjectContentInput,
) -> S3Result<Answered<SelectObjectContentOutput>> {
let read_principal = SseKmsPrincipal::from_request(&envelope);
let validation = validate_select_request(envelope.headers(), &mut input)?;
log_select_request_summary(&input, &validation);
let query_timeout = s3_select_query_timeout();
let query_deadline = Instant::now() + query_timeout;
let engine_input = Arc::new(select_input_from_s3s(input.clone()).map_err(|err| {
S3Error::with_message(S3ErrorCode::InternalError, format!("convert the SelectObjectContent request: {err}"))
})?);
let input = Arc::new(input);
let db = timeout_at(query_deadline, current_s3select_db((*engine_input).clone(), false))
let db = timeout_at(query_deadline, current_s3select_db((*input).clone(), false))
.await
.map_err(|_| select_query_timeout_error(query_timeout.as_secs()))?
.map_err(map_query_error_to_s3)?;
let admission = db.try_reserve_query().map_err(map_query_error_to_s3)?;
let (snapshot, response_headers) = timeout_at(
query_deadline,
prepare_select_object_snapshot(&req.headers, &input, read_principal.as_ref()),
prepare_select_object_snapshot(envelope.headers(), &input, read_principal.as_ref()),
)
.await
.map_err(|_| select_query_timeout_error(query_timeout.as_secs()))??;
@@ -146,8 +146,13 @@ pub async fn execute_select_object_content(
return Err(map_select_error_to_s3(&SelectError::TruncatedInput));
}
let snapshot = Arc::new(snapshot);
let query =
Query::new_with_snapshot(Context { input: engine_input }, input.request.expression.clone(), Arc::clone(&snapshot));
let query = Query::new_with_snapshot(
Context {
input: Arc::clone(&input),
},
input.request.expression.clone(),
Arc::clone(&snapshot),
);
let query_handle = timeout_at(query_deadline, db.execute_admitted(&query, admission))
.await
.map_err(|_| select_query_timeout_error(query_timeout.as_secs()))?
@@ -167,10 +172,6 @@ pub async fn execute_select_object_content(
.clone()
.try_reserve_owned()
.map_err(|_| map_select_error_to_s3(&SelectError::InternalError))?;
let mut response = S3Response::new(SelectObjectContentOutput {
payload: Some(SelectObjectContentEventStream::new(ReceiverStream::new(rx))),
});
response.headers = response_headers;
spawn_traced(async move {
send_select_events_until_deadline(
output,
@@ -188,7 +189,19 @@ pub async fn execute_select_object_content(
.await;
});
Ok(response)
Ok(Answered {
output: select_output(rx),
status: None,
headers: response_headers,
})
}
fn select_output(rx: mpsc::Receiver<S3Result<SelectObjectContentEvent>>) -> SelectObjectContentOutput {
let events = ReceiverStream::new(rx)
.map(|item| item.map_err(|error| SelectEventError::new(error.code().as_str(), error.message().unwrap_or_default())));
SelectObjectContentOutput {
payload: Some(SelectEventStream::new(events)),
}
}
async fn send_select_events_until_deadline<L: SelectSnapshotFence>(
@@ -604,20 +617,20 @@ fn validate_select_request(headers: &http::HeaderMap, input: &mut SelectObjectCo
if input.request.expression.len() > MAX_SELECT_EXPRESSION_BYTES {
return Err(S3Error::new(S3ErrorCode::ExpressionTooLong));
}
let Some(expression_type) = canonical_enum(input.request.expression_type.as_str(), &[ExpressionType::SQL]) else {
if !input
.request
.expression_type
.as_str()
.eq_ignore_ascii_case(ExpressionType::SQL.as_str())
{
return Err(S3Error::new(S3ErrorCode::InvalidExpressionType));
};
input.request.expression_type = ExpressionType::from_static(expression_type);
}
input.request.expression_type = ExpressionType::SQL;
normalize_input_serialization(&mut input.request.input_serialization)?;
let compressed_input = is_compressed_input(&input.request.input_serialization);
if compressed_input
&& input.request.scan_range.as_ref().is_some_and(|range| {
is_noop_scan_range(&rustfs_s3select_api::request::ScanRange {
start: range.start,
end: range.end,
})
})
&& input.request.scan_range.as_ref().is_some_and(is_noop_scan_range)
&& !input
.request
.input_serialization
@@ -656,16 +669,15 @@ fn is_compressed_input(input: &InputSerialization) -> bool {
input
.compression_type
.as_ref()
.is_some_and(|compression| compression.as_str() != CompressionType::NONE)
.is_some_and(|compression| *compression != CompressionType::NONE)
}
/// Matches an S3 enum value in any ASCII case, as MinIO does, and returns the
/// canonical spelling that downstream code compares against.
fn canonical_enum(value: &str, allowed: &[&'static str]) -> Option<&'static str> {
fn canonical_enum<T>(value: &T, allowed: impl IntoIterator<Item = T>, as_str: impl Fn(&T) -> &str) -> Option<T> {
allowed
.iter()
.copied()
.find(|candidate| candidate.eq_ignore_ascii_case(value))
.into_iter()
.find(|candidate| as_str(candidate).eq_ignore_ascii_case(as_str(value)))
}
fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()> {
@@ -680,8 +692,9 @@ fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()>
if let Some(compression) = input.compression_type.as_mut() {
let Some(canonical) = canonical_enum(
compression.as_str(),
&[CompressionType::NONE, CompressionType::GZIP, CompressionType::BZIP2],
compression,
[CompressionType::NONE, CompressionType::GZIP, CompressionType::BZIP2],
CompressionType::as_str,
) else {
return Err(map_select_error_to_s3(&SelectError::InvalidCompressionFormat));
};
@@ -691,26 +704,23 @@ fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()>
INVALID_REQUEST_PARAMETER_MESSAGE,
));
}
*compression = CompressionType::from_static(canonical);
*compression = canonical;
}
input
.compression_type
.get_or_insert_with(|| CompressionType::from_static(CompressionType::NONE));
input.compression_type.get_or_insert(CompressionType::NONE);
if let Some(csv) = input.csv.as_mut() {
let file_header_info = csv
.file_header_info
.get_or_insert_with(|| FileHeaderInfo::from_static(FileHeaderInfo::NONE));
let file_header_info = csv.file_header_info.get_or_insert(FileHeaderInfo::NONE);
let Some(canonical) = canonical_enum(
file_header_info.as_str(),
&[FileHeaderInfo::NONE, FileHeaderInfo::USE, FileHeaderInfo::IGNORE],
file_header_info,
[FileHeaderInfo::NONE, FileHeaderInfo::USE, FileHeaderInfo::IGNORE],
FileHeaderInfo::as_str,
) else {
return Err(S3Error::with_message(
S3ErrorCode::InvalidFileHeaderInfo,
"The FileHeaderInfo value is not valid. Only NONE, USE, and IGNORE are supported.",
));
};
*file_header_info = FileHeaderInfo::from_static(canonical);
*file_header_info = canonical;
validate_single_byte(csv.comments.as_deref(), S3ErrorCode::InvalidRequestParameter)?;
validate_single_character(csv.quote_character.as_deref())?;
validate_single_character(csv.quote_escape_character.as_deref())?;
@@ -719,11 +729,11 @@ fn normalize_input_serialization(input: &mut InputSerialization) -> S3Result<()>
}
if let Some(json) = input.json.as_mut() {
let json_type = json.type_.get_or_insert_with(|| JSONType::from_static(JSONType::LINES));
let Some(canonical) = canonical_enum(json_type.as_str(), &[JSONType::DOCUMENT, JSONType::LINES]) else {
let json_type = json.r#type.get_or_insert(JsonType::LINES);
let Some(canonical) = canonical_enum(json_type, [JsonType::DOCUMENT, JsonType::LINES], JsonType::as_str) else {
return Err(S3Error::new(S3ErrorCode::InvalidJsonType));
};
*json_type = JSONType::from_static(canonical);
*json_type = canonical;
}
Ok(())
@@ -744,10 +754,11 @@ fn normalize_output_serialization(output: &mut OutputSerialization) -> S3Result<
validate_single_byte(csv.quote_escape_character.as_deref(), S3ErrorCode::InvalidRequestParameter)?;
validate_output_record_delimiter(csv.record_delimiter.as_deref())?;
if let Some(quote_fields) = csv.quote_fields.as_mut() {
let Some(canonical) = canonical_enum(quote_fields.as_str(), &[QuoteFields::ALWAYS, QuoteFields::ASNEEDED]) else {
let Some(canonical) = canonical_enum(quote_fields, [QuoteFields::ALWAYS, QuoteFields::ASNEEDED], QuoteFields::as_str)
else {
return Err(S3Error::new(S3ErrorCode::InvalidQuoteFields));
};
*quote_fields = QuoteFields::from_static(canonical);
*quote_fields = canonical;
}
return Ok(SelectOutputFormat::Csv(csv.clone()));
}
@@ -863,7 +874,7 @@ async fn prepare_select_object_snapshot(
let opts = get_opts(&input.bucket, &input.key, None, None, headers)
.await
.map_err(ApiError::from)?;
let store = get_validated_store(&input.bucket).await?;
let store = get_validated_store_owned(&input.bucket).await?;
let snapshot = store
.prepare_select_object_snapshot(&input.bucket, &input.key, headers, &opts)
.await
@@ -941,7 +952,7 @@ impl SelectOutputEncoder {
}
#[cfg(test)]
fn encode_csv_batch(batch: &RecordBatch, config: &CSVOutput, buffer: &mut BytesMut) -> S3Result<()> {
fn encode_csv_batch(batch: &RecordBatch, config: &CsvOutput, buffer: &mut BytesMut) -> S3Result<()> {
if encode_csv_batch_limited(batch, 0..batch.num_rows(), config, buffer, usize::MAX)? == batch.num_rows() {
Ok(())
} else {
@@ -952,7 +963,7 @@ fn encode_csv_batch(batch: &RecordBatch, config: &CSVOutput, buffer: &mut BytesM
fn encode_csv_batch_limited(
batch: &RecordBatch,
rows: Range<usize>,
config: &CSVOutput,
config: &CsvOutput,
buffer: &mut BytesMut,
max_bytes: usize,
) -> S3Result<usize> {
@@ -965,7 +976,7 @@ fn encode_csv_batch_limited(
let quote_all = config
.quote_fields
.as_ref()
.is_some_and(|quote_fields| quote_fields.as_str() == QuoteFields::ALWAYS);
.is_some_and(|quote_fields| *quote_fields == QuoteFields::ALWAYS);
let mut output = LimitedBytesWriter::new(buffer, max_bytes);
let mut field = BoundedText::default();
let mut encoded_rows = 0;
@@ -1431,7 +1442,7 @@ fn write_limited(output: &mut LimitedBytesWriter<'_>, bytes: &[u8]) -> bool {
output.write_all(bytes).is_ok()
}
fn json_record_delimiter(config: &JSONOutput) -> &[u8] {
fn json_record_delimiter(config: &JsonOutput) -> &[u8] {
if let Some(delimiter) = config.record_delimiter.as_deref() {
delimiter.as_bytes()
} else {
@@ -1440,7 +1451,7 @@ fn json_record_delimiter(config: &JSONOutput) -> &[u8] {
}
#[cfg(test)]
fn encode_json_batch(batch: &RecordBatch, config: &JSONOutput, buffer: &mut BytesMut) -> S3Result<()> {
fn encode_json_batch(batch: &RecordBatch, config: &JsonOutput, buffer: &mut BytesMut) -> S3Result<()> {
if encode_json_batch_limited(batch, 0..batch.num_rows(), config, buffer, usize::MAX)? == batch.num_rows() {
Ok(())
} else {
@@ -1451,7 +1462,7 @@ fn encode_json_batch(batch: &RecordBatch, config: &JSONOutput, buffer: &mut Byte
fn encode_json_batch_limited(
batch: &RecordBatch,
rows: Range<usize>,
config: &JSONOutput,
config: &JsonOutput,
buffer: &mut BytesMut,
max_bytes: usize,
) -> S3Result<usize> {
@@ -1653,10 +1664,8 @@ fn select_query_timeout_error(seconds: u64) -> S3Error {
map_query_error_to_s3(SelectError::QueryTimeout { seconds }.into())
}
fn is_json_document(json: &JSONInput) -> bool {
json.type_
.as_ref()
.is_some_and(|json_type| json_type.as_str() == JSONType::DOCUMENT)
fn is_json_document(json: &JsonInput) -> bool {
json.r#type.as_ref().is_some_and(|json_type| *json_type == JsonType::DOCUMENT)
}
#[cfg(test)]
@@ -1677,8 +1686,9 @@ mod tests {
physical_plan::stream::RecordBatchStreamAdapter,
sql::sqlparser::parser::ParserError,
};
use rustfs_gateway_types::compat::s3s_5761ddfe::select::output_to_s3s;
use rustfs_s3select_api::request::{CsvInput, ParquetInput, ScanRange};
use rustfs_test_utils::TestECStoreEnv;
use s3s::dto::{CSVInput, ParquetInput, ScanRange};
fn event_stream_headers(mut bytes: &[u8]) -> Vec<Vec<(String, String)>> {
let mut messages = Vec::new();
@@ -1717,7 +1727,9 @@ mod tests {
}
async fn http_xml_error(error: S3Error) -> (StatusCode, String) {
let response = error.to_http_response().expect("S3 error should serialize to HTTP");
let response = s3s::S3Error::from(error)
.to_http_response()
.expect("S3 error should serialize to HTTP");
let status = response.status();
let body = http_body_util::BodyExt::collect(response.into_body())
.await
@@ -1817,18 +1829,18 @@ mod tests {
let mut input = base_input();
input.bucket = bucket.clone();
input.key = "input.csv".to_owned();
let request = S3Request {
input,
method: http::Method::POST,
uri: format!("/{bucket}/input.csv?select&select-type=2").parse().expect("URI"),
headers: HeaderMap::new(),
extensions: http::Extensions::new(),
credentials: None,
region: None,
service: None,
trailing_headers: None,
};
let response = execute_select_object_content(request)
let envelope = RequestEnvelope::from_parts(
crate::app::RequestHead {
method: http::Method::POST,
uri: format!("/{bucket}/input.csv?select&select-type=2").parse().expect("URI"),
headers: HeaderMap::new(),
credentials: None,
region: None,
service: None,
},
http::Extensions::new(),
);
let response = execute_select_object_content(envelope, input)
.await
.expect("start actual SELECT response");
// Keep the response alive without polling its event stream. The
@@ -1875,15 +1887,15 @@ mod tests {
sse_customer_key_md5: None,
request: SelectObjectContentRequest {
expression: "SELECT * FROM s3object".to_string(),
expression_type: ExpressionType::from_static(ExpressionType::SQL),
expression_type: ExpressionType::SQL,
input_serialization: InputSerialization {
csv: Some(CSVInput::default()),
csv: Some(CsvInput::default()),
compression_type: None,
json: None,
parquet: None,
},
output_serialization: OutputSerialization {
csv: Some(CSVOutput::default()),
csv: Some(CsvOutput::default()),
json: None,
},
request_progress: None,
@@ -1894,7 +1906,7 @@ mod tests {
fn csv_validation() -> SelectValidation {
SelectValidation {
output_format: SelectOutputFormat::Csv(CSVOutput::default()),
output_format: SelectOutputFormat::Csv(CsvOutput::default()),
progress_enabled: false,
reports_input_metrics: true,
}
@@ -2062,9 +2074,7 @@ mod tests {
StatusCode::BAD_REQUEST,
),
(
SelectError::InvalidCompressionFormatForObject {
compression: CompressionType::GZIP,
},
SelectError::InvalidCompressionFormatForObject { compression: "GZIP" },
S3ErrorCode::InvalidCompressionFormat,
StatusCode::BAD_REQUEST,
),
@@ -2279,7 +2289,7 @@ mod tests {
let values = builder.finish();
let schema = Arc::new(Schema::new(vec![Field::new("value", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows =
@@ -2298,7 +2308,7 @@ mod tests {
let values = DictionaryArray::<Int32Type>::try_new(keys, dictionary).expect("test dictionary should be valid");
let schema = Arc::new(Schema::new(vec![Field::new("value", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer).expect("referenced empty values should batch");
@@ -2318,7 +2328,7 @@ mod tests {
DictionaryArray::<Int32Type>::try_new(Int32Array::from(keys), dictionary).expect("test dictionary should be valid");
let schema = Arc::new(Schema::new(vec![Field::new("value", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows =
@@ -2334,7 +2344,7 @@ mod tests {
let values = StringArray::from(vec!["x"; row_count]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows =
@@ -2350,7 +2360,7 @@ mod tests {
let values = ListArray::from_iter_primitive::<Int32Type, _, _>([Some([Some(1)]), Some([Some(2)])]);
let schema = Arc::new(Schema::new(vec![Field::new("items", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer).expect("small nested JSON rows should encode");
@@ -2371,7 +2381,7 @@ mod tests {
let values = builder.finish();
let schema = Arc::new(Schema::new(vec![Field::new("items", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer).expect("the complete first row should encode");
@@ -2393,7 +2403,7 @@ mod tests {
let values = StringArray::from(vec!["ok", quoted.as_str()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
let encoded_rows =
@@ -2413,7 +2423,7 @@ mod tests {
]));
let batch =
RecordBatch::try_new(schema, vec![Arc::new(values), Arc::new(nested)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::from(b"staged\n".as_slice());
let error = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer)
@@ -2434,7 +2444,7 @@ mod tests {
let values = builder.finish();
let schema = Arc::new(Schema::new(vec![Field::new("items", values.data_type().clone(), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let error = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer)
@@ -2472,7 +2482,7 @@ mod tests {
]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(profile), Arc::new(labels), Arc::new(dictionary), Arc::new(run)])
.expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
assert_eq!(
@@ -2490,7 +2500,7 @@ mod tests {
let values = NullArray::new(1);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Null, true)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
assert_eq!(
@@ -2507,7 +2517,7 @@ mod tests {
let values = DictionaryArray::<Int32Type>::try_new(keys, dictionary).expect("test dictionary should be valid");
let schema = Arc::new(Schema::new(vec![Field::new("value", values.data_type().clone(), true)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
assert_eq!(
@@ -2523,7 +2533,7 @@ mod tests {
.expect("test dictionary should be valid");
let schema = Arc::new(Schema::new(vec![Field::new("value", values.data_type().clone(), true)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
assert_eq!(
@@ -2539,7 +2549,7 @@ mod tests {
let values = NullArray::new(1);
let schema = Arc::new(Schema::new(vec![Field::new(field_name, DataType::Null, true)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let error = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer)
@@ -2556,7 +2566,7 @@ mod tests {
let values = StructArray::new_null(vec![child].into(), 1);
let schema = Arc::new(Schema::new(vec![Field::new("nested", values.data_type().clone(), true)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
assert_eq!(
@@ -2572,7 +2582,7 @@ mod tests {
let values = StringArray::from(vec!["a", "b"]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput {
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput {
record_delimiter: Some(delimiter.clone()),
}));
let mut buffer = BytesMut::new();
@@ -2589,7 +2599,7 @@ mod tests {
let values = StringArray::from(vec![value.as_str()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
let error =
@@ -2610,7 +2620,7 @@ mod tests {
.map(|_| Arc::clone(&values) as datafusion::arrow::array::ArrayRef)
.collect::<Vec<_>>();
let batch = RecordBatch::try_new(Arc::new(Schema::new(fields)), columns).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
let error = encode_batch_turn(&mut encoder, &batch, 0, &mut buffer)
@@ -2627,7 +2637,7 @@ mod tests {
let values = StringArray::from(vec![value.as_str()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CSVOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Csv(CsvOutput::default()));
let mut buffer = BytesMut::new();
if excess == 0 {
@@ -2653,7 +2663,7 @@ mod tests {
let values = StringArray::from(vec![value.as_str()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
if excess == 0 {
@@ -2681,7 +2691,7 @@ mod tests {
let values = BinaryArray::from(vec![value.as_slice()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Binary, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
if excess == 0 {
@@ -2715,7 +2725,7 @@ mod tests {
let values = StringArray::from(vec![value.as_str()]);
let schema = Arc::new(Schema::new(vec![Field::new("value", DataType::Utf8, false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(values)]).expect("test record batch should be valid");
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JSONOutput::default()));
let mut encoder = SelectOutputEncoder::new(SelectOutputFormat::Json(JsonOutput::default()));
let mut buffer = BytesMut::new();
if excess == 0 {
@@ -3108,7 +3118,10 @@ mod tests {
let (producer, rx, lease_released) = spawn_test_producer(output, 4);
producer.await.expect("producer should finish successfully");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut byte_stream = output_to_s3s(select_output(rx))
.payload
.expect("Select stream")
.into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
@@ -3234,7 +3247,10 @@ mod tests {
let (producer, rx, lease_released) = spawn_test_producer(output, 2);
producer.await.expect("producer should emit the terminal Select error");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut byte_stream = output_to_s3s(select_output(rx))
.payload
.expect("Select stream")
.into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
@@ -3278,7 +3294,7 @@ mod tests {
assert!(sql_body.contains("<Message>"));
let mut input = base_input();
input.request.input_serialization.compression_type = Some(CompressionType::from_static("SNAPPY"));
input.request.input_serialization.compression_type = Some(CompressionType::custom("SNAPPY"));
let compression_error =
validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown compression must fail before streaming");
let (compression_status, compression_body) = http_xml_error(compression_error).await;
@@ -3314,7 +3330,10 @@ mod tests {
.await
.expect("producer should emit records followed by the terminal error");
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut byte_stream = output_to_s3s(select_output(rx))
.payload
.expect("Select stream")
.into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("event-stream message should serialize"));
@@ -3378,39 +3397,45 @@ mod tests {
#[tokio::test(start_paused = true)]
async fn producer_drops_query_stream_and_snapshot_lease_when_receiver_closes() {
let (stream_dropped_tx, stream_dropped_rx) = tokio::sync::oneshot::channel::<()>();
let output = Box::pin(RecordBatchStreamAdapter::new(
Arc::new(Schema::empty()),
futures::stream::once(async move {
let _stream_dropped = stream_dropped_tx;
futures::future::pending::<Result<RecordBatch, DataFusionError>>().await
}),
));
let (event_channel, rx) = test_event_channel(2);
let (lease, lease_released) = lease_drop_signal();
let producer = send_select_events_until_deadline(
output,
event_channel,
csv_validation(),
Arc::new(SelectInputMetrics::default()),
Instant::now() + Duration::from_secs(300),
300,
lease,
);
tokio::pin!(producer);
for legacy in [false, true] {
let (stream_dropped_tx, stream_dropped_rx) = tokio::sync::oneshot::channel::<()>();
let output = Box::pin(RecordBatchStreamAdapter::new(
Arc::new(Schema::empty()),
futures::stream::once(async move {
let _stream_dropped = stream_dropped_tx;
futures::future::pending::<Result<RecordBatch, DataFusionError>>().await
}),
));
let (event_channel, rx) = test_event_channel(2);
let (lease, lease_released) = lease_drop_signal();
let producer = send_select_events_until_deadline(
output,
event_channel,
csv_validation(),
Arc::new(SelectInputMetrics::default()),
Instant::now() + Duration::from_secs(300),
300,
lease,
);
tokio::pin!(producer);
assert!(futures::poll!(producer.as_mut()).is_pending());
drop(rx);
assert!(futures::poll!(producer.as_mut()).is_pending());
if legacy {
drop(output_to_s3s(select_output(rx)));
} else {
drop(select_output(rx));
}
assert!(
futures::poll!(producer.as_mut()).is_ready(),
"producer should observe the closed receiver"
);
assert!(
stream_dropped_rx.await.is_err(),
"query stream should be dropped when the receiver closes"
);
assert!(lease_released.await.is_ok(), "receiver close should release the snapshot lease");
assert!(
futures::poll!(producer.as_mut()).is_ready(),
"producer should observe the closed receiver"
);
assert!(
stream_dropped_rx.await.is_err(),
"query stream should be dropped when the receiver closes"
);
assert!(lease_released.await.is_ok(), "receiver close should release the snapshot lease");
}
}
#[tokio::test(start_paused = true)]
@@ -3535,26 +3560,17 @@ mod tests {
.input_serialization
.csv
.as_ref()
.and_then(|csv| csv.file_header_info.as_ref())
.map(|value| value.as_str()),
Some(FileHeaderInfo::NONE)
);
assert_eq!(
input
.request
.input_serialization
.compression_type
.as_ref()
.map(|value| value.as_str()),
Some(CompressionType::NONE)
.and_then(|csv| csv.file_header_info.as_ref()),
Some(&FileHeaderInfo::NONE)
);
assert_eq!(input.request.input_serialization.compression_type.as_ref(), Some(&CompressionType::NONE));
}
#[test]
fn validate_preserves_supported_compression_for_csv_and_json_lines() {
for compression in [CompressionType::GZIP, CompressionType::BZIP2] {
let mut csv_input = base_input();
csv_input.request.input_serialization.compression_type = Some(CompressionType::from_static(compression));
csv_input.request.input_serialization.compression_type = Some(compression.clone());
validate_select_request(&HeaderMap::new(), &mut csv_input).expect("compressed CSV should be accepted");
assert_eq!(
csv_input
@@ -3563,15 +3579,15 @@ mod tests {
.compression_type
.as_ref()
.map(|value| value.as_str()),
Some(compression)
Some(compression.as_str())
);
let mut json_input = base_input();
json_input.request.input_serialization.csv = None;
json_input.request.input_serialization.json = Some(JSONInput {
type_: Some(JSONType::from_static(JSONType::LINES)),
json_input.request.input_serialization.json = Some(JsonInput {
r#type: Some(JsonType::LINES),
});
json_input.request.input_serialization.compression_type = Some(CompressionType::from_static(compression));
json_input.request.input_serialization.compression_type = Some(compression.clone());
validate_select_request(&HeaderMap::new(), &mut json_input).expect("compressed JSON LINES should be accepted");
assert_eq!(
json_input
@@ -3580,7 +3596,7 @@ mod tests {
.compression_type
.as_ref()
.map(|value| value.as_str()),
Some(compression)
Some(compression.as_str())
);
}
}
@@ -3590,7 +3606,7 @@ mod tests {
let mut input = base_input();
input.request.input_serialization.csv = None;
input.request.input_serialization.parquet = Some(ParquetInput {});
input.request.input_serialization.compression_type = Some(CompressionType::from_static(CompressionType::GZIP));
input.request.input_serialization.compression_type = Some(CompressionType::GZIP);
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("compressed Parquet must fail");
@@ -3601,7 +3617,7 @@ mod tests {
#[test]
fn validate_normalizes_noop_compressed_scan_range_and_rejects_real_ranges() {
let mut noop = base_input();
noop.request.input_serialization.compression_type = Some(CompressionType::from_static(CompressionType::GZIP));
noop.request.input_serialization.compression_type = Some(CompressionType::GZIP);
noop.request.scan_range = Some(ScanRange {
start: Some(0),
end: None,
@@ -3610,7 +3626,7 @@ mod tests {
assert!(noop.request.scan_range.is_none());
let mut ranged = base_input();
ranged.request.input_serialization.compression_type = Some(CompressionType::from_static(CompressionType::GZIP));
ranged.request.input_serialization.compression_type = Some(CompressionType::GZIP);
ranged.request.scan_range = Some(ScanRange {
start: Some(1),
end: None,
@@ -3622,7 +3638,7 @@ mod tests {
assert_eq!(error.message(), Some("Scan range queries are not supported on this type of object."));
let mut malformed = base_input();
malformed.request.input_serialization.compression_type = Some(CompressionType::from_static(CompressionType::GZIP));
malformed.request.input_serialization.compression_type = Some(CompressionType::GZIP);
malformed.request.scan_range = Some(ScanRange {
start: Some(10),
end: Some(1),
@@ -3683,7 +3699,7 @@ mod tests {
.as_mut()
.expect("CSV input")
.allow_quoted_record_delimiter = Some(true);
input.request.input_serialization.compression_type = Some(CompressionType::from_static(compression));
input.request.input_serialization.compression_type = Some(compression.clone());
input.request.scan_range = Some(range);
let error = validate_select_request(&HeaderMap::new(), &mut input)
.expect_err("quoted record delimiters cannot be combined with ScanRange");
@@ -3725,7 +3741,7 @@ mod tests {
.csv
.as_mut()
.expect("base input should use CSV")
.file_header_info = Some(FileHeaderInfo::from_static("INVALID"));
.file_header_info = Some(FileHeaderInfo::custom("INVALID"));
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown header mode must fail");
@@ -3740,8 +3756,8 @@ mod tests {
fn json_input(json_type: &str) -> SelectObjectContentInput {
let mut input = base_input();
input.request.input_serialization.csv = None;
input.request.input_serialization.json = Some(JSONInput {
type_: Some(JSONType::from(json_type.to_string())),
input.request.input_serialization.json = Some(JsonInput {
r#type: Some(JsonType::custom(json_type.to_string())),
});
input
}
@@ -3754,7 +3770,7 @@ mod tests {
("none", CompressionType::NONE),
] {
let mut input = base_input();
input.request.input_serialization.compression_type = Some(CompressionType::from(value.to_string()));
input.request.input_serialization.compression_type = Some(CompressionType::custom(value.to_string()));
validate_select_request(&HeaderMap::new(), &mut input).expect("CompressionType should match any case");
assert_eq!(
input
@@ -3763,11 +3779,11 @@ mod tests {
.compression_type
.as_ref()
.map(|v| v.as_str()),
Some(expected)
Some(expected.as_str())
);
}
for (value, expected) in [("lines", JSONType::LINES), ("Document", JSONType::DOCUMENT)] {
for (value, expected) in [("lines", JsonType::LINES), ("Document", JsonType::DOCUMENT)] {
let mut input = json_input(value);
validate_select_request(&HeaderMap::new(), &mut input).expect("JSON Type should match any case");
let json_type = input
@@ -3775,8 +3791,8 @@ mod tests {
.input_serialization
.json
.as_ref()
.and_then(|json| json.type_.as_ref());
assert_eq!(json_type.map(|v| v.as_str()), Some(expected));
.and_then(|json| json.r#type.as_ref());
assert_eq!(json_type.map(|v| v.as_str()), Some(expected.as_str()));
}
let mut input = json_input("document");
validate_select_request(&HeaderMap::new(), &mut input).expect("lowercase DOCUMENT should be accepted");
@@ -3794,10 +3810,10 @@ mod tests {
.csv
.as_mut()
.expect("base input should use CSV")
.file_header_info = Some(FileHeaderInfo::from(value.to_string()));
.file_header_info = Some(FileHeaderInfo::custom(value.to_string()));
validate_select_request(&HeaderMap::new(), &mut input).expect("FileHeaderInfo should match any case");
let csv = input.request.input_serialization.csv.as_ref().expect("CSV input");
assert_eq!(csv.file_header_info.as_ref().map(|v| v.as_str()), Some(expected));
assert_eq!(csv.file_header_info.as_ref().map(|v| v.as_str()), Some(expected.as_str()));
}
for (value, expected) in [("always", QuoteFields::ALWAYS), ("AsNeeded", QuoteFields::ASNEEDED)] {
@@ -3808,25 +3824,25 @@ mod tests {
.csv
.as_mut()
.expect("base output should use CSV")
.quote_fields = Some(QuoteFields::from(value.to_string()));
.quote_fields = Some(QuoteFields::custom(value.to_string()));
let validation = validate_select_request(&HeaderMap::new(), &mut input).expect("QuoteFields should match any case");
let SelectOutputFormat::Csv(csv) = validation.output_format else {
panic!("CSV output expected");
};
assert_eq!(csv.quote_fields.as_ref().map(|v| v.as_str()), Some(expected));
assert_eq!(csv.quote_fields.as_ref().map(|v| v.as_str()), Some(expected.as_str()));
}
let mut input = base_input();
input.request.expression_type = ExpressionType::from("sql".to_string());
input.request.expression_type = ExpressionType::custom("sql".to_string());
validate_select_request(&HeaderMap::new(), &mut input).expect("ExpressionType should match any case");
assert_eq!(input.request.expression_type.as_str(), ExpressionType::SQL);
assert_eq!(input.request.expression_type, ExpressionType::SQL);
}
#[test]
fn validate_still_rejects_unknown_enum_values_in_any_case() {
for value in ["gzipx", "snappy", " GZIP"] {
let mut input = base_input();
input.request.input_serialization.compression_type = Some(CompressionType::from(value.to_string()));
input.request.input_serialization.compression_type = Some(CompressionType::custom(value.to_string()));
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown compression must fail");
assert_eq!(error.code(), &S3ErrorCode::InvalidCompressionFormat, "{value}");
}
@@ -3842,7 +3858,7 @@ mod tests {
.csv
.as_mut()
.expect("base input should use CSV")
.file_header_info = Some(FileHeaderInfo::from("used".to_string()));
.file_header_info = Some(FileHeaderInfo::custom("used".to_string()));
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown header mode must fail");
assert_eq!(error.code(), &S3ErrorCode::InvalidFileHeaderInfo);
@@ -3853,12 +3869,12 @@ mod tests {
.csv
.as_mut()
.expect("base output should use CSV")
.quote_fields = Some(QuoteFields::from("never".to_string()));
.quote_fields = Some(QuoteFields::custom("never".to_string()));
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown QuoteFields must fail");
assert_eq!(error.code(), &S3ErrorCode::InvalidQuoteFields);
let mut input = base_input();
input.request.expression_type = ExpressionType::from("sqlite".to_string());
input.request.expression_type = ExpressionType::custom("sqlite".to_string());
let error = validate_select_request(&HeaderMap::new(), &mut input).expect_err("unknown ExpressionType must fail");
assert_eq!(error.code(), &S3ErrorCode::InvalidExpressionType);
}
@@ -3989,7 +4005,7 @@ mod tests {
writer.finish().expect("Arrow JSON reference should finish");
}
let mut actual = BytesMut::new();
encode_json_batch(&batch, &JSONOutput::default(), &mut actual).expect("S3 Select JSON should encode");
encode_json_batch(&batch, &JsonOutput::default(), &mut actual).expect("S3 Select JSON should encode");
assert_eq!(actual.as_ref(), expected.as_slice(), "type: {}", batch.column(0).data_type());
}
@@ -4010,7 +4026,7 @@ mod tests {
.unwrap();
let mut bytes = BytesMut::new();
encode_json_batch(&batch, &JSONOutput::default(), &mut bytes).unwrap();
encode_json_batch(&batch, &JsonOutput::default(), &mut bytes).unwrap();
assert_eq!(bytes.as_ref(), b"{\"name\":\"a\"}\n{\"name\":\"b\"}\n");
}
@@ -4033,7 +4049,7 @@ mod tests {
let mut bytes = BytesMut::new();
encode_json_batch(
&batch,
&JSONOutput {
&JsonOutput {
record_delimiter: Some("|".to_string()),
},
&mut bytes,
@@ -4088,7 +4104,7 @@ mod tests {
let mut bytes = BytesMut::new();
encode_csv_batch(
&batch,
&CSVOutput {
&CsvOutput {
field_delimiter: Some("|".to_string()),
record_delimiter: Some("\r\n".to_string()),
..Default::default()
@@ -4108,9 +4124,9 @@ mod tests {
encode_csv_batch(
&batch,
&CSVOutput {
&CsvOutput {
quote_escape_character: Some("\\".to_string()),
quote_fields: Some(QuoteFields::from_static(QuoteFields::ASNEEDED)),
quote_fields: Some(QuoteFields::ASNEEDED),
record_delimiter: Some("|".to_string()),
..Default::default()
},
@@ -4140,10 +4156,10 @@ mod tests {
encode_csv_batch(
&batch,
&CSVOutput {
&CsvOutput {
quote_character: Some("'".to_string()),
quote_escape_character: Some("\\".to_string()),
quote_fields: Some(QuoteFields::from_static(QuoteFields::ALWAYS)),
quote_fields: Some(QuoteFields::ALWAYS),
record_delimiter: Some("|".to_string()),
..Default::default()
},
@@ -4214,7 +4230,10 @@ mod tests {
.expect("test channel should accept Records");
drop(tx);
let mut byte_stream = SelectObjectContentEventStream::new(ReceiverStream::new(rx)).into_byte_stream();
let mut byte_stream = output_to_s3s(select_output(rx))
.payload
.expect("Select stream")
.into_byte_stream();
let mut encoded = Vec::new();
while let Some(chunk) = byte_stream.next().await {
encoded.extend_from_slice(&chunk.expect("Records event should serialize"));
@@ -4235,8 +4254,8 @@ mod tests {
let mut input = base_input();
input.request.input_serialization = InputSerialization {
csv: None,
json: Some(JSONInput {
type_: Some(JSONType::from_static(JSONType::DOCUMENT)),
json: Some(JsonInput {
r#type: Some(JsonType::DOCUMENT),
}),
parquet: None,
compression_type: None,
@@ -4256,8 +4275,8 @@ mod tests {
let mut input = base_input();
input.request.input_serialization = InputSerialization {
csv: None,
json: Some(JSONInput {
type_: Some(JSONType::from_static(JSONType::LINES)),
json: Some(JsonInput {
r#type: Some(JsonType::LINES),
}),
parquet: None,
compression_type: None,
+66 -6
View File
@@ -12,13 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! SelectObjectContent requests across the legacy S3 edge.
//! SelectObjectContent requests from either S3 edge.
//!
//! Responsible for: converting the s3s request the edge decoded into the
//! request the query engine runs (rustfs/backlog#2746), through the gateway's
//! s3s seam for the serialization shapes. Not responsible for: validating the
//! request; the select use case does that before the engine is built.
//! DELETE BY T4.1 (rustfs/backlog#2783) with the rest of the legacy edge.
//! Both S3 edges pass the request in the shape the query engine reads.
//! The select use case validates it before building the engine.
//! The s3s conversion is deleted with the legacy edge (rustfs/backlog#2783).
use rustfs_gateway_types::compat::ConversionError;
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::shapes::input_serialization::input_serialization_from_s3s;
@@ -69,6 +67,43 @@ pub(crate) fn select_input_from_s3s(
})
}
/// Moves the gateway's flat request into the shape the query engine reads.
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) fn select_input_from_gateway(
input: rustfs_gateway_types::dto::SelectObjectContentInput,
) -> engine::SelectObjectContentInput {
let rustfs_gateway_types::dto::SelectObjectContentInput {
bucket,
key,
expected_bucket_owner,
sse_customer_algorithm,
sse_customer_key,
sse_customer_key_md5,
expression,
expression_type,
input_serialization,
output_serialization,
request_progress,
scan_range,
} = input;
engine::SelectObjectContentInput {
bucket: bucket.as_str().to_owned(),
key: key.as_str().to_owned(),
expected_bucket_owner,
sse_customer_algorithm,
sse_customer_key: sse_customer_key.map(|key| key.expose_secret().to_owned()),
sse_customer_key_md5,
request: engine::SelectObjectContentRequest {
expression,
expression_type,
input_serialization,
output_serialization,
request_progress,
scan_range,
},
}
}
#[cfg(test)]
mod tests {
use super::select_input_from_s3s;
@@ -159,6 +194,31 @@ mod tests {
assert_eq!((range.start, range.end), (Some(3), Some(9)));
}
#[test]
fn gateway_projection_preserves_the_query_and_read_parameters() {
let source = input(InputSerialization {
csv: Some(CSVInput::default()),
..Default::default()
});
let owned = rustfs_gateway_types::compat::s3s_5761ddfe::select::input_from_s3s(source, &Default::default())
.expect("materialize the test request");
let converted = super::select_input_from_gateway(owned);
assert_eq!((converted.bucket.as_str(), converted.key.as_str()), ("bucket", "key"));
assert_eq!(converted.expected_bucket_owner.as_deref(), Some("owner"));
assert_eq!(converted.sse_customer_algorithm.as_deref(), Some("AES256"));
assert_eq!(converted.sse_customer_key.as_deref(), Some("key-text"));
assert_eq!(converted.sse_customer_key_md5.as_deref(), Some("client-md5"));
let request = converted.request;
assert_eq!(request.expression, "SELECT * FROM S3Object");
assert_eq!(request.expression_type, super::engine::ExpressionType::SQL);
assert!(request.input_serialization.csv.is_some());
let csv = request.output_serialization.csv.expect("CSV output");
assert_eq!(csv.field_delimiter.as_deref(), Some(";"));
assert_eq!(request.request_progress.and_then(|progress| progress.enabled), Some(true));
let range = request.scan_range.expect("scan range");
assert_eq!((range.start, range.end), (Some(3), Some(9)));
}
#[test]
fn keeps_the_json_type_and_an_unknown_expression_type() {
let mut source = input(InputSerialization {
+42 -27
View File
@@ -127,15 +127,14 @@ pub(crate) mod data_usage {
pub(crate) use crate::storage::storage_api::ecstore_data_usage::quota_object_size;
#[cfg(test)]
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
pub(crate) async fn load_admin_data_usage_from_backend_cached(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
}
pub(crate) async fn load_bucket_usage_from_backend_cached(
@@ -335,11 +334,11 @@ pub(crate) mod access {
pub(crate) use crate::storage::storage_api::access_consumer::ReqInfo;
pub(crate) use crate::storage::storage_api::access_consumer::requested_bucket_force_delete;
pub(crate) use crate::storage::storage_api::access_consumer::{
AuthorizationRequest, RequestedObjectLegalHold, RequestedObjectLockRetention, TABLE_DATA_PLANE_LIST_CURSOR_PREFIX,
TableDataPlaneListAccess, TableDataPlaneListCursorPosition, TableDataPlanePublicationGuards,
apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard, authorize_copy_object_tags, authorize_request,
bucket_config_mutation_incarnation, delete_object_authorize_action, has_bypass_governance_header,
load_bucket_generation_from_store, log_list_buckets_iam_implicit_deny, odm_read_generation,
AuthorizationRequest, CopySourceObject, RequestedObjectLegalHold, RequestedObjectLockRetention,
TABLE_DATA_PLANE_LIST_CURSOR_PREFIX, TableDataPlaneListAccess, TableDataPlaneListCursorPosition,
TableDataPlanePublicationGuards, apply_bucket_generation_guard, apply_copy_source_bucket_generation_guard,
authorize_copy_object_tags, authorize_request, bucket_config_mutation_incarnation, delete_object_authorize_action,
has_bypass_governance_header, load_bucket_generation_from_store, log_list_buckets_iam_implicit_deny, odm_read_generation,
prepare_list_buckets_iam_authorization, prepare_odm_read_generation, recursive_force_delete_has_authenticated_caller,
replication_request_authorized, req_info_mut, req_info_ref, retained_table_data_plane_publication_guards,
set_requested_object_lock_retain_until,
@@ -347,7 +346,7 @@ pub(crate) mod access {
}
pub(crate) mod bucket {
use crate::storage::storage_api::ecstore_bucket::lifecycle::bucket_lifecycle_ops::{
pub(crate) use crate::storage::storage_api::ecstore_bucket::lifecycle::bucket_lifecycle_ops::{
RestoreOutputLocation, RestoreRequest, RestoreS3Location, RestoreTag, RestoreTagging,
};
use std::sync::Arc;
@@ -394,7 +393,7 @@ pub(crate) mod bucket {
crate::storage::storage_api::ecstore_bucket::lifecycle::lifecycle::Lifecycle::predict_expiration(lifecycle, obj).await
}
// Remove this conversion when RestoreObject receives Gateway inputs.
// The legacy edge keeps values that its decoder permits before domain validation.
pub(crate) fn restore_request_from_wire(request: s3s::dto::RestoreRequest) -> RestoreRequest {
use rustfs_gateway_types::dto;
@@ -945,6 +944,10 @@ pub(crate) mod bucket {
}
pub(crate) mod metadata {
pub(crate) use crate::storage::storage_api::ecstore_bucket::metadata::{MAX_BUCKET_TAGS, validate_tag_set_entries};
pub(crate) use crate::storage::storage_api::{
BUCKET_ACCELERATE_CONFIG, BUCKET_LOGGING_CONFIG, BUCKET_REQUEST_PAYMENT_CONFIG, BUCKET_WEBSITE_CONFIG,
};
pub(crate) const BUCKET_CORS_CONFIG: &str = crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_CORS_CONFIG;
pub(crate) const BUCKET_LIFECYCLE_CONFIG: &str =
crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_LIFECYCLE_CONFIG;
@@ -959,11 +962,13 @@ pub(crate) mod bucket {
pub(crate) const BUCKET_TARGETS_FILE: &str = crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_TARGETS_FILE;
pub(crate) const BUCKET_VERSIONING_CONFIG: &str =
crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_VERSIONING_CONFIG;
#[cfg(test)]
pub(crate) const OBJECT_LOCK_CONFIG: &str = crate::storage::storage_api::ecstore_bucket::metadata::OBJECT_LOCK_CONFIG;
}
pub(crate) mod metadata_sys {
pub(crate) use crate::storage::storage_api::ecstore_bucket::metadata_sys::{
get_accelerate_config, get_logging_config, get_request_payment_config, get_website_config,
};
use std::sync::Arc;
use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration, PersistedTagging};
@@ -1633,13 +1638,14 @@ pub(crate) mod helper {
MutationCompletionProbe, capture_mutation_event_for_test, pause_committed_mutation_for_test,
};
pub(crate) use crate::storage::storage_api::helper_consumer::{
OperationHelper, build_event_resp_elements, build_owned_event_resp_elements, run_owned_mutation,
spawn_background_with_context,
OperationHelper, build_owned_event_resp_elements, run_owned_mutation, spawn_background_with_context,
};
}
pub(crate) mod object_utils {
pub(crate) use crate::storage::storage_api::{to_s3s_etag, to_s3s_xml_etag};
#[cfg(test)]
pub(crate) use crate::storage::storage_api::to_s3s_etag;
pub(crate) use crate::storage::storage_api::to_s3s_xml_etag;
}
pub(crate) mod io {
@@ -1710,14 +1716,13 @@ pub(crate) mod s3_api {
pub(crate) mod multipart {
pub(crate) use crate::storage::storage_api::s3_api_consumer::multipart::{
ListMultipartUploadsParams, build_list_parts_output, parse_list_multipart_uploads_params, parse_list_parts_params,
parse_upload_part_number,
ListMultipartUploadsParams, parse_list_multipart_uploads_params, parse_list_parts_params, parse_upload_part_number,
};
}
pub(crate) mod tagging {
pub(crate) use crate::storage::storage_api::s3_api_consumer::tagging::{
parse_copy_object_tags, resolve_copy_object_tags, validate_bucket_tag_set,
parse_copy_object_tags, resolve_copy_object_tags,
};
}
}
@@ -1754,8 +1759,8 @@ pub(crate) mod bucket_usecase {
pub(crate) use super::{access, bucket, error, helper, object_utils, request_context, s3, s3_api, sse};
pub(crate) use crate::storage::storage_api::{
ECStore, StorageObjectInfo, StorageObjectOptions, get_validated_store, process_lambda_configurations,
process_queue_configurations, process_topic_configurations, validate_list_object_unordered_with_delimiter,
ECStore, StorageObjectInfo, StorageObjectOptions, get_validated_store, validate_list_object_unordered_with_delimiter,
validate_website_configuration,
};
}
@@ -1818,12 +1823,12 @@ pub(crate) mod object_usecase {
}
}
#[cfg(test)]
pub(crate) use super::ecfs;
pub(crate) use super::{
access, bucket, compression, concurrency, data_usage, deadlock_detector, error, head_prefix, helper, io, object_utils,
options, request_context, s3_api, set_disk, sse, storage_class, timeout_wrapper,
access, bucket, compression, concurrency, data_usage, deadlock_detector, error, head_prefix, helper, io, options,
request_context, s3_api, set_disk, sse, storage_class, timeout_wrapper,
};
#[cfg(test)]
pub(crate) use super::{ecfs, object_utils};
pub(crate) use crate::storage::storage_api::{
ECStore, GetObjectReader, OldCurrentSize, RFC1123, StorageDeletedObject, StorageObjectInfo,
StorageObjectLockDeleteOptions, StorageObjectOptions, StorageObjectToDelete, StoragePutObjReader, check_preconditions,
@@ -1839,11 +1844,17 @@ pub(crate) mod object_usecase {
pub(crate) mod object_meta_usecase {
pub(crate) mod contract {
pub(crate) use super::super::storage_contracts::object::ObjectLockRetentionOptions;
pub(crate) use super::super::storage_contracts::{BucketOperations, BucketOptions, ObjectOperations};
}
pub(crate) use crate::storage::storage_api::check_retention_for_modification;
pub(crate) use crate::storage::storage_api::ecfs_consumer::object_lock::validate_bucket_object_lock_enabled;
pub(crate) use crate::storage::storage_api::ecstore_bucket::tagging::decode_tags as decode_stored_tags;
pub(crate) use crate::storage::storage_api::ecstore_bucket::metadata::validate_tag_set_entries;
pub(crate) use crate::storage::storage_api::ecstore_bucket::tagging::{
decode_tags as decode_stored_tags, encode_tags as encode_stored_tags,
};
pub(crate) use crate::storage::storage_api::ecstore_bucket::versioning::VersioningApi;
pub(crate) use crate::storage::storage_api::parse_object_version_id;
}
@@ -1856,9 +1867,13 @@ pub(crate) mod multipart_usecase {
}
pub(crate) mod multipart {
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) use super::super::super::storage_contracts::multipart::ListPartsOwner;
pub(crate) use super::super::super::storage_contracts::{
CompletePart, ListMultipartsInfo, MultipartInfo, MultipartOperations, MultipartUploadResult,
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations, MultipartUploadResult,
};
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) use super::super::super::storage_contracts::{MultipartAbortAuthorization, MultipartOwner};
pub(crate) use crate::storage::storage_api::s3_api_consumer::multipart::contract::multipart::MAX_MULTIPART_PART_NUMBER;
}
@@ -1886,8 +1901,8 @@ pub(crate) mod select_object {
#[cfg(test)]
pub(crate) use crate::storage::storage_api::StorageError;
pub(crate) use crate::storage::storage_api::{
StoragePrepareSelectObjectSnapshotError, StorageSelectObjectSnapshot, get_validated_store, validate_sse_headers_for_read,
validate_ssec_for_read,
StoragePrepareSelectObjectSnapshotError, StorageSelectObjectSnapshot, get_validated_store_owned,
validate_sse_headers_for_read, validate_ssec_for_read,
};
}

Some files were not shown because too many files have changed in this diff Show More