Compare commits

..

22 Commits

Author SHA1 Message Date
cxymds 0517566fad chore: sync main for scanner statistics review 2026-11-15 15:56:57 +08:00
Hauser Zhi 8f361e3a6f perf(ecstore): share rename fanout metadata (#9432)
* perf(ecstore): share rename fanout metadata

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): stabilize async regression fixtures

Run the multipart refusal matrix on the shared large-stack harness, wait for scanner metadata commit tails before restart discovery, and observe io_uring owner release alongside in-flight retirement.

Keep the production paths unchanged while preserving each original failure oracle.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:05:20 +08:00
Hauser Zhi 3cee9c154b perf(grpc): attribute rebuilds and bound bulk responses (#9420)
* perf(grpc): attribute rebuilds and bound bulk responses

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(grpc): attribute rebuild success after publication

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(obs): add exact HTTP/2 ping timeout diagnostics

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(multipart): run refusal matrix on large stack

Run the real-store completion refusal matrix through the shared 32 MiB test harness so its deep async storage future does not overflow libtest's default stack.

Keep the original test name and all refusal-order assertions intact.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* chore(deps): pin nixl-sys workspace source

Manage the nixl-sys Git revision from the workspace dependency table and point the optional Linux transport dependency at houseme/nixl commit cc636c83f13d692e85b67e55aed092413d16fb41.

Allow the reviewed source in cargo-deny and refresh the lockfile, replacing the unmaintained bincode dependency with the fork's wincode graph.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): authenticate msgpack peer capability

Add an optional boot-epoch-bound HMAC capability proof and cache the verified per-peer observation without enabling any msgpack send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(grpc): bind peer capability to connection generations (#9433)

* feat(grpc): track physical connection generations

Observe every tonic physical reconnect through a stable custom connector while preserving existing TCP, TLS, HTTP/2, cache, and public channel behavior.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* feat(rpc): bind msgpack capability to connections

Scope authenticated msgpack observations to channel identity and physical connection generation so transparent reconnects require a fresh proof without enabling the send policy.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:05:09 +08:00
Ramakrishna Chilaka 504cc06d0c perf(ecstore): stage inline rollback backups and fuse backup reclaim (#9445)
Stage inline overwrite rollback backups (xl.meta.bkp) directly relative
to the pinned destination directory descriptor in RenameCommitGuard
instead of staging in .rustfs.sys/tmp and walking both paths a second
time. Also fuse post-commit xl.meta.bkp removal and empty-ancestor
directory pruning into a single blocking dispatch while stopping at live
object roots.
2026-11-15 15:04:18 +08:00
Hauser Zhi 44145ef327 chore(deps): update flake.lock (#9441)
Flake lock file updates:

• Updated input 'nixpkgs':
    'github:NixOS/nixpkgs/4fe25e1' (2026-11-06)
  → 'github:NixOS/nixpkgs/2074cc6' (2026-11-12)
• Updated input 'rust-overlay':
    'github:oxalica/rust-overlay/d3fe742' (2026-11-07)
  → 'github:oxalica/rust-overlay/5b87ee0' (2026-11-14)
2026-11-15 15:04:11 +08:00
Hauser Zhi ba9e8342f8 test(storage): add fail-closed quiescence gate (#9436)
* test(storage): add fail-closed quiescence gate

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(storage): gate benchmark rounds on quiescence

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(bench): bound quiescence Docker calls

Start the storage-quiescence deadline before Docker discovery, pass the remaining budget to every subprocess, and reject samples completed after the deadline.

Cover delayed initial lookup and delayed exec through the supported Docker binary override.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-11-15 15:04:04 +08:00
Hauser Zhi a41019802d test(storage): expose direct-write validation controls (#9430)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:53 +08:00
Hauser Zhi 4313812e24 perf(ecstore): keep inline commits fused with file grouping (#9427)
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 15:03:45 +08:00
hector 5c58a56c4c fix: recognize gVisor directfs bind mounts in the FS guard (#9426)
gVisor presents a host bind mount as 9p, so statfs reports V9FS.
When the mountinfo options include both trans=fd and directfs, the
startup filesystem guard treats that endpoint as local POSIX storage
and logs that the host type is not visible. Real 9p and virtio shares
still warn.

Co-authored-by: hector <majinghe@users.noreply.github.com>
Co-authored-by: Hauser Zhi <housemecn@gmail.com>
2026-11-15 15:03:37 +08:00
GatewayJ 8196603a76 fix(iam): authorize delegated service account creation (#9408)
* fix(iam): authorize delegated service account creation

* docs(iam): clarify explicit grants for custom administrator policies

* fix(iam): include service account delegation in admin wildcards

* style(policy): format delegated authorization test cases

* fix(iam): keep delegated authorization and replication consistent

* fix(ci): include delegated account test in E2E selections
2026-11-15 15:03:24 +08:00
Hauser Zhi 1e62cbaeaf perf(ecstore): integrate safe sync and allocator diagnostics (#9428)
* feat(diagnostics): restore allocator and runtime metrics

Port the missing live-byte accounting and serving-runtime observations from 13a30abca. Reuse the current MiMalloc GlobalAlloc wrapper, expose explicit metric availability, and cover allocation operations and feature selection.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): detach queued fdatasync followers safely

Adapt the follower timeout experiment from 138bc69f6 with exclusive queue ownership transfer. Move queued paths and resource units into independent sync, retain claimed batch ownership through completion, and record observed queue time. Cover publication before the original batch resumes, sync failures, claimed batches, and queue accounting.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(ecstore): index sync followers and isolate missing sources

Use reusable slab slots and monotonic identities for constant-time queued follower extraction. Avoid retaining follower group references when the timeout is disabled. Preserve claimed-batch ownership and reject stale tickets after slot reuse.

Keep missing-source results local to their requests so healthy requests still synchronize their own files. Preserve fail-fast behavior for other disk I/O failures. Repair failure-probe synchronization and exercise source disappearance, reused slots, disabled tracking, cancellation and bounded admission on Linux.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ci): apply storage stack to multipart refusal fixture

Route the real-store absent-binding refusal-order fixture through the existing bounded 4 MiB storage setup in both default and CI profiles. The Linux CI failure reproduced with the default 2 MiB stack. Quota, legal-hold, error, cause, audit and notification assertions execute through the established storage fixture budget.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): make sync worker idle handoff atomic

Move batch claim, in-flight accounting and completion onto the existing worker guard. Publish the idle state under the queue lock so an enqueue during cleanup establishes successor-worker ownership. Keep stale cleanup conditional on current ownership.

Add a controlled interleaving regression that detaches the final queued follower and enqueues a successor after the empty claim. The old worker leaves that request without an owner; the refactored worker hands it off and completes one physical sync.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(ecstore): await grouped sync resource release

Wait for native producer admission units to reach zero after the blocking sync finishes. This removes a scheduler-dependent assertion window exposed by the latest main integration while preserving the cancellation and ownership checks.

Co-Authored-By: heihutu <heihutu@gmail.com>

Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-11-15 14:58:54 +08:00
Hauser Zhi da1817ae01 test(rpc): expose msgpack fleet confirmation (#9429) 2026-11-15 14:54:23 +08:00
cxymds 73895b37e8 test(admin): bind snapshot fixtures to current bucket identities 2026-11-15 14:49:54 +08:00
overtrue c2b5fbb138 feat(gateway): serve CreateMultipartUpload through the owned path
Share authorization between both CreateMultipartUpload edges. Keep the
decoded legacy fields intact, preserve the replication destination
version and apply the captured bucket incarnation before creating the
upload. Leave checksum, encryption and retention metadata on the same
storage path.

Verify the frozen legacy behavior on 2c2af51a before the migration and
exercise both stacks with 66 refusal cases and 22 success controls.
Read back sessions, version, retention and encryption metadata.

Fault probes made authorization tests fail on both edges when an action
or the resource binding was lost. Storage tests detected a different
destination version and unexpected sessions when generation, quota, KMS
or SSE checks were removed. The wire and
helper tests failed for early name rejection, lost timestamp precision,
metadata, checksum, SSE response, the operation count and unexpected
notification.

On 77da6339, 185 scoped tests passed with a 32 MiB test stack. The default
stack run passed 184 tests and reproduced the existing overflow in
completion_absent_binding_refusal_order_baseline. Both all-target
Clippy variants, test wiring, script tests and prepush passed. After
integration with 97a7639f, gateway all-target check and prepush passed.
Whole-workspace CI and rio-v2 were outside the local verification.
2026-11-15 14:08:38 +08:00
overtrue 97a7639f59 feat(s3): serve UploadPart through the gateway
Share the streaming multipart write path and staged authorization across both edges. Resolve the native upload ID against the active storage record at the existing lookup. Carry checksums, trailers, SSE values and the stored ETag through each response projection.

Add seven tests for application lookup, route authorization, framing and durable part state. Exercise 62 refusals and 10 successful requests across those fixtures. The existing SSE-C completion test also covers the migrated UploadPart core.

Verification on 1467d32695: fixed legacy contract, 147 gateway-feature tests and 119 default tests, both all-target Clippy legs, script tests, test wiring and prepush checks. The test scopes include the entire access module. The gateway scope includes PutObject and both refusal matrices.

Twenty-two local fault controls exercised edge authorization, policy and generation, resource and upload ID projection, owner binding, lookup count and error ordering, logical length, trailer publication, checksum validation and typed producer errors. Each fault failed its intended test after the same artifact passed with the selector unset.

Linux RDMA and a native KMS denial through UploadPart remain unverified.
2026-11-15 13:41:36 +08:00
cxymds 00243c7e71 chore: sync main for scanner validation
Resolve the shared checkpoint fixture update by retaining production drain and the stronger carried-value, certificate, budget and acknowledgement assertions. Preserve upstream functionality and the validated scanner implementation.
2026-11-15 12:02:04 +08:00
cxymds 36bc1057ac fix(ci): give the multipart refusal fixture its storage stack 2026-11-15 11:25:22 +08:00
cxymds 32f7f70a29 feat(scanner): observe bucket reuse and production proof gates 2026-11-15 11:25:22 +08:00
cxymds 5d32827af5 fix(usage): validate cached statistics against bucket identity 2026-11-15 11:25:06 +08:00
cxymds 04ef836aa3 fix(scanner): cap superseded retries at the nominal cycle 2026-11-15 11:25:05 +08:00
cxymds 03850c1cd6 fix(scanner): preserve baselines during backup catchup
Require decoded-primary provenance before accepting an unchanged baseline after its backup catches up. Keep current-revision CAS and publication fences, and align the real checkpoint fixture with the bounded-cycle carry contract.
2026-11-15 11:24:42 +08:00
cxymds 654be96186 feat(scanner): expose scan work and reuse observations 2026-11-14 21:25:19 +08:00
107 changed files with 8643 additions and 3288 deletions
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=8fef326a3fddd8a548e2cbcc6ea8159f831113595fb003eeebbd9aeb4ac23e48
sha256-linux=8b4381578377704ac8b1b5156128b83f036ec4aedc9d863fff25df15112420fc
sha256-darwin=24a8cfaa1b9125bd69d7d4a7ff752a8ab8ed350c761d90ae8472ef52393e2686
sha256-linux=014257197fbbfbb48cab2864da9e10d96ee877d478fa29d119f8186532ea293d
+1 -1
View File
@@ -1 +1 @@
sha256=061d0a697967d2dc8f472ff885886e069b1cf4aeaa5f230e177b7236017c23b6
sha256=fd5683273bba1eae3b4402a4e854a5d168ace845bd0e5fc801f8e336485a4394
-2
View File
@@ -39,8 +39,6 @@ script-tests: ## Run shell script tests
./scripts/test_entrypoint_directories.sh
./scripts/test_internode_grpc_ab_bench.sh
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_grpc_window.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_internode_msgpack_peer_canary.py
./scripts/test_internode_msgpack_peer_canary_runner.sh
./scripts/test_object_batch_bench_enhanced.sh
./scripts/test_package_service_scripts.sh
./scripts/test_hotpath_warp_ab_gate.sh
+3 -2
View File
@@ -141,8 +141,9 @@ setup = 'ecstore-large-stack'
filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
setup = 'ecstore-base-stack'
# The multipart refusal fixture composes real-store setup, abort, quota and legal-hold futures.
[[profile.default.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
@@ -465,7 +466,7 @@ filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::complete::tests::completion_absent_binding_refusal_order_baseline|app::multipart_usecase::complete::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
@@ -35,6 +35,7 @@ services:
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
@@ -46,6 +47,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -59,7 +62,7 @@ services:
- node1_data_2:/data/rustfs2
- node1_data_3:/data/rustfs3
ports:
- "9000:9000"
- "${RUSTFS_NODE1_PORT:-9000}:9000"
networks:
- rustfs-cluster-net
@@ -85,6 +88,7 @@ services:
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
@@ -96,6 +100,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -109,7 +115,7 @@ services:
- node2_data_2:/data/rustfs2
- node2_data_3:/data/rustfs3
ports:
- "9001:9000"
- "${RUSTFS_NODE2_PORT:-9001}:9000"
networks:
- rustfs-cluster-net
@@ -135,6 +141,7 @@ services:
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
@@ -146,6 +153,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -159,7 +168,7 @@ services:
- node3_data_2:/data/rustfs2
- node3_data_3:/data/rustfs3
ports:
- "9002:9000"
- "${RUSTFS_NODE3_PORT:-9002}:9000"
networks:
- rustfs-cluster-net
@@ -185,6 +194,7 @@ services:
- RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
- RUSTFS_DURABILITY_MODE=${RUSTFS_DURABILITY_MODE:-strict}
- RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE=${RUSTFS_OBJECT_DIRECT_IO_WRITE_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE:-false}
- RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS=${RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS:-0}
# Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
@@ -196,6 +206,8 @@ services:
- RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
- RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
- RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED:-false}
- RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED=${RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED:-false}
- RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
- RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
- RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
@@ -209,7 +221,7 @@ services:
- node4_data_2:/data/rustfs2
- node4_data_3:/data/rustfs3
ports:
- "9003:9000"
- "${RUSTFS_NODE4_PORT:-9003}:9000"
networks:
- rustfs-cluster-net
Generated
+1
View File
@@ -10187,6 +10187,7 @@ dependencies = [
"serial_test",
"sha2 0.11.1",
"shadow-rs",
"slab",
"smallvec",
"temp-env",
"tempfile",
+1
View File
@@ -186,6 +186,7 @@ memchr = "2.8.3"
minlz = "1.2.3"
reqwest = "0.13.5"
rustfs-kafka-async = { version = "1.4.1" }
slab = { version = "0.4.12" }
socket2 = { version = "0.6.5" }
tokio = { version = "1.54.1" }
tokio-rustls = { default-features = false, version = "0.26.6" }
+2
View File
@@ -302,6 +302,8 @@ Drive timeout profile preset:
RustFS production guidance remains direct-attached local POSIX filesystems. Network-mounted filesystems (for example `nfs`, `cifs`, `smb2`, and `fuse.*`) are treated as unsupported by this startup guard. On Windows, endpoints on network drives (UNC shares such as `\\server\share` and mapped drive letters) are treated the same way, even though Windows reports the server's filesystem (usually `NTFS`) for them.
A local POSIX filesystem bind-mounted into a gVisor sandbox with directfs is a supported storage path. gVisor presents that bind mount as `9p`/`V9FS`, so the host filesystem type is not visible; the startup guard recognizes mountinfo options `trans=fd` and `directfs` and does not treat the mount as unsupported. Genuine 9p, virtio, and NFS shares remain unsupported.
## 📄 License
This project is licensed under the Apache License 2.0 - see the [LICENSE](../../LICENSE) file for details.
-9
View File
@@ -121,17 +121,9 @@ pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY: bool = false;
pub const ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED: bool = false;
/// Permit current per-peer capability proof to replace fleet-wide confirmation.
///
/// This remains ineffective unless [`ENV_INTERNODE_RPC_MSGPACK_ONLY`] is also enabled. Unknown,
/// unsupported, revoked, or stale connection generations keep dual-writing JSON.
pub const ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: &str = "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY";
pub const DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY: bool = false;
// Compile-time invariants: dual-write by default so the base build is byte-for-byte legacy behavior.
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED);
const _: () = assert!(!DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY);
/// Require target-bound v2 signatures on every internode gRPC request, rejecting the legacy
/// constant-target fallback instead of accepting it (<https://github.com/rustfs/backlog/issues/1327>).
@@ -533,7 +525,6 @@ mod tests {
ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED,
"RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED"
);
assert_eq!(ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY, "RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY");
}
#[test]
@@ -171,6 +171,70 @@ impl DataUsageBucketObservation {
}
impl DataUsageInfo {
/// Hide values that cannot be bound to the current storage-owned identity.
///
/// `incarnations` contains canonical non-nil identities read from storage;
/// a missing entry means identity was unavailable. Neither legacy values
/// without identity nor metadata read failures establish confirmed zero.
/// Changed responses are display-only and cannot authorize quota or writes.
/// If retained counters overflow, all values remain unknown.
/// Returns the number of canonical values removed.
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
let original_count = self.buckets_usage.len();
let original_observation_count = self.usage_snapshot_bucket_observations.len();
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
incarnations.get(bucket).is_some_and(|current| {
!current.is_empty()
&& current != "00000000-0000-0000-0000-000000000000"
&& usage.bucket_incarnation.as_ref() == Some(current)
})
};
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
matches_current(bucket, &observation.usage)
&& observation
.sources
.iter()
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
});
if original_count == self.buckets_usage.len()
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
{
return 0;
}
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
self.buckets_usage.clear();
self.usage_snapshot_bucket_observations.clear();
[0; 4]
});
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_admin_projection = true;
original_count.saturating_sub(self.buckets_usage.len())
}
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
@@ -321,3 +321,110 @@ fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
#[test]
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
let mut candidate = snapshot();
let original = candidate.clone();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
assert_eq!(candidate, original);
assert!(candidate.is_complete_bucket_usage_snapshot());
}
#[test]
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
for identity in [
None,
Some(""),
Some("corrupt"),
Some("00000000-0000-0000-0000-000000000000"),
Some("22222222-2222-4222-8222-222222222222"),
] {
let mut candidate = snapshot();
candidate.usage_snapshot_bucket_observations.clear();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = identity.map(str::to_string);
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
1,
"identity {identity:?}"
);
assert!(!candidate.buckets_usage.contains_key("bucket"));
assert!(!candidate.bucket_sizes.contains_key("bucket"));
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
assert!(!candidate.is_valid_partial_snapshot());
}
for current in ["", "00000000-0000-0000-0000-000000000000"] {
let mut candidate = snapshot();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some(current.to_string());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
1
);
assert!(candidate.usage_snapshot_admin_projection);
}
let mut candidate = snapshot();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(7));
candidate.buckets_count = 2;
candidate.bucket_sizes.insert("cold".to_string(), 7);
candidate.calculate_totals();
candidate.replication_info.insert("target".to_string(), Default::default());
candidate.tier_stats = Some(Default::default());
candidate.unknown_tier_stats = Some(Default::default());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
1
);
assert_eq!(candidate.buckets_usage.len(), 1);
assert_eq!(candidate.buckets_usage["cold"].size, 7);
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
assert_eq!(
(
candidate.buckets_count,
candidate.objects_total_size,
candidate.objects_total_count,
candidate.versions_total_count
),
(1, 7, 7, 7)
);
assert!(candidate.replication_info.is_empty());
assert!(candidate.tier_stats.is_none());
assert!(candidate.unknown_tier_stats.is_none());
assert!(candidate.usage_snapshot_admin_projection);
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_overflow_leaves_totals_unknown() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
candidate.buckets_usage.insert("small".to_string(), usage(1));
candidate.buckets_count = 3;
let current = HashMap::from([
("large".to_string(), INCARNATION.to_string()),
("small".to_string(), INCARNATION.to_string()),
]);
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&current), 3);
assert!(candidate.buckets_usage.is_empty());
assert!(candidate.bucket_sizes.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
}
+133 -1
View File
@@ -146,7 +146,11 @@ async fn create_service_account_for(
Some(serde_json::json!({ "targetUser": parent }).to_string()),
)
.await?;
let response: serde_json::Value = serde_json::from_str(&response)?;
service_account_keys(&response)
}
fn service_account_keys(response: &str) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
let response: serde_json::Value = serde_json::from_str(response)?;
let access_key = response["credentials"]["accessKey"]
.as_str()
.ok_or("service account response should contain credentials.accessKey")?
@@ -344,6 +348,134 @@ async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestRe
Ok(())
}
#[tokio::test]
async fn test_delegated_service_account_creation_through_console_sessions() -> TestResult {
init_logging();
let mut env = RustFSTestEnvironment::new().await?;
env.start_rustfs_server_without_cleanup(vec![]).await?;
let bucket = "delegated-backups";
let target = "backup-user";
let operator = "backup-operator";
let operator_secret = "operatorSecret123";
let wildcard_admin = "backup-admin";
let wildcard_secret = "adminSecret123";
let s3 = env.create_s3_client();
s3.create_bucket().bucket(bucket).send().await?;
for (name, secret) in [
(target, "backupSecret123"),
(operator, operator_secret),
(wildcard_admin, wildcard_secret),
("other-user", "otherUserSecret123"),
] {
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-user?accessKey={name}"),
Some(serde_json::json!({"secretKey":secret,"status":"enabled"}).to_string()),
)
.await?;
}
let delegation = serde_json::json!({
"Version":"2012-10-17",
"Statement":[
{"Effect":"Allow","Action":"admin:CreateServiceAccount"},
{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:s3:::*"},
{"Effect":"Allow","Action":"admin:CreateServiceAccountForUser","Condition":{"StringEquals":{"svc:TargetUser":target}}}
]
});
let wildcard = serde_json::json!({
"Version":"2012-10-17",
"Statement":[
{"Effect":"Allow","Action":"admin:*"},
{"Effect":"Allow","Action":"sts:AssumeRole","Resource":"arn:aws:s3:::*"}
]
});
for (user, name, policy) in [
(wildcard_admin, "backup-admin-wildcard", wildcard.to_string()),
(target, "backup-writer", bucket_rw_policy(bucket)),
(operator, "backup-key-manager", delegation.to_string()),
] {
admin_ok(
&env,
http::Method::PUT,
&format!("/rustfs/admin/v3/add-canned-policy?name={name}"),
Some(policy),
)
.await?;
admin_ok(
&env,
http::Method::POST,
"/rustfs/admin/v3/idp/builtin/policy/attach",
Some(serde_json::json!({"policies":[name],"user":user}).to_string()),
)
.await?;
}
for (name, secret) in [
(operator, operator_secret),
(wildcard_admin, wildcard_secret),
(env.access_key.as_str(), env.secret_key.as_str()),
] {
let session = build_test_sts_client(&env.url, name, secret, None, "delegated-create")
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/console")
.role_session_name("console")
.send()
.await?;
let caller = session.credentials().ok_or("missing console session")?;
let create = |user: &str| {
admin_request_with_session_token(
&env.url,
http::Method::PUT,
"/rustfs/admin/v3/add-service-account",
Some(serde_json::json!({"targetUser":user}).to_string()),
caller.access_key_id(),
caller.secret_access_key(),
Some(caller.session_token()),
)
};
let (status, response) = create(target).await?;
assert_eq!(status, StatusCode::OK, "create for target as {name}");
let (key, secret) = service_account_keys(&response)?;
let client = build_s3_client(&env.url, &key, &secret);
wait_for_s3_put(&client, bucket, "backup", Duration::from_secs(10)).await?;
let denied = client
.create_bucket()
.bucket("outside-backup-policy")
.send()
.await
.expect_err("target cannot create unrelated buckets");
assert_eq!(denied.as_service_error().and_then(|e| e.code()), Some("AccessDenied"));
if name != env.access_key {
let (status, _) = create(&env.access_key).await?;
assert_eq!(status, StatusCode::FORBIDDEN, "delegated targets exclude root");
}
if name == operator {
let (status, _) = create("other-user").await?;
assert_eq!(status, StatusCode::FORBIDDEN, "outside delegated target");
}
}
let restricted = build_test_sts_client(&env.url, &env.access_key, &env.secret_key, None, "restricted-root")
.assume_role()
.role_arn("arn:aws:iam::123456789012:role/console")
.role_session_name("restricted-console")
.policy(r#"{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":"admin:CreateServiceAccount"}]}"#)
.send()
.await?;
let caller = restricted.credentials().ok_or("missing restricted root session")?;
let (status, _) = admin_request_with_session_token(
&env.url,
http::Method::PUT,
"/rustfs/admin/v3/add-service-account",
Some(serde_json::json!({"targetUser":target}).to_string()),
caller.access_key_id(),
caller.secret_access_key(),
Some(caller.session_token()),
)
.await?;
assert_eq!(status, StatusCode::FORBIDDEN, "restricted root session requires delegated permission");
Ok(())
}
/// Full user -> policy -> service-account lifecycle, proving each management
/// call takes effect on the data plane, not just that the endpoint answers 200.
#[tokio::test]
+1
View File
@@ -126,6 +126,7 @@ test-util = []
e2e-test-hooks = []
[dependencies]
slab.workspace = true
hashbrown.workspace = true
hotpath.workspace = true
rustfs-filemeta.workspace = true
+2 -2
View File
@@ -453,8 +453,8 @@ pub mod data_usage {
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_compression_total_from_memory,
load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
+7 -586
View File
@@ -55,7 +55,7 @@ use std::{
task::{Context, Poll},
time::Duration,
};
use tonic::{codegen::StdError, service::interceptor::InterceptedService};
use tonic::service::interceptor::InterceptedService;
use tower::Service;
use tracing::debug;
use uuid::Uuid;
@@ -84,7 +84,7 @@ pub(crate) async fn scanner_control_time_out_client(
let connection_state = channel.connection_state();
let interceptor = interceptor.with_connection_state(connection_state.clone());
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
BatchLockDeadlineChannel::new(channel),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -104,7 +104,7 @@ pub async fn heal_control_time_out_client(
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::HEAL_CONTROL_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
BatchLockDeadlineChannel::new(channel),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -123,7 +123,7 @@ pub async fn tier_mutation_control_time_out_client(
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::TIER_MUTATION_RPC_MAX_MESSAGE_SIZE;
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
BatchLockDeadlineChannel::new(channel),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -150,7 +150,7 @@ pub async fn node_service_time_out_client_for_class(
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
BatchLockDeadlineChannel::new(channel),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -159,39 +159,6 @@ pub async fn node_service_time_out_client_for_class(
.max_encoding_message_size(max_message_size))
}
/// Build an eligible request client and capture its peer capability exactly once for this RPC.
pub(crate) async fn node_service_time_out_client_for_class_with_msgpack_policy(
addr: &str,
interceptor: TonicInterceptor,
class: ChannelClass,
) -> Result<
(
NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>,
MsgpackSendPolicy,
),
Box<dyn Error>,
> {
let interceptor = interceptor.with_rpc_audience(addr)?;
let audience = interceptor.replay_scope_audience();
let channel = get_internode_channel_for_class(addr, class).await?;
let connection_state = channel.connection_state();
let policy =
prepare_msgpack_send_policy(audience.as_deref(), connection_state.as_ref().and_then(GrpcConnectionState::snapshot));
let interceptor = interceptor.with_connection_state(connection_state.clone());
let max_message_size = rustfs_protos::internode_rpc_max_message_size();
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
audience,
connection_state,
);
Ok((
NodeServiceClient::with_interceptor(channel, interceptor)
.max_decoding_message_size(max_message_size)
.max_encoding_message_size(max_message_size),
policy,
))
}
pub(crate) async fn node_service_delete_recovery_client(
addr: &str,
interceptor: TonicInterceptor,
@@ -204,7 +171,7 @@ pub(crate) async fn node_service_delete_recovery_client(
let connection_state = channel.connection_state();
let interceptor = interceptor.with_connection_state(connection_state.clone());
let channel = ReplayScopeChannel::new(
ConnectionGenerationChannel::new(BatchLockDeadlineChannel::new(channel), connection_state.clone()),
BatchLockDeadlineChannel::new(channel),
interceptor.replay_scope_audience(),
connection_state,
);
@@ -324,238 +291,7 @@ pub struct ReplayScopeChannel<S> {
}
/// The channel type used by internode clients after v2 authentication and replay-scope handling.
pub type AuthenticatedChannel = ReplayScopeChannel<ConnectionGenerationChannel<BatchLockDeadlineChannel<InternodeChannel>>>;
const CONNECTION_GENERATION_PENDING: u8 = 0;
const CONNECTION_GENERATION_DISPATCHED: u8 = 1;
const CONNECTION_GENERATION_REJECTED: u8 = 2;
#[derive(Debug)]
struct ConnectionGenerationGuardInner {
expected: GrpcConnectionSnapshot,
expected_boot_epoch: Option<Uuid>,
outcome: AtomicU8,
}
/// A request-owned proof that its encoded body targets one physical connection generation.
#[derive(Clone, Debug)]
pub(crate) struct ConnectionGenerationGuard(Arc<ConnectionGenerationGuardInner>);
impl ConnectionGenerationGuard {
pub(crate) fn new(expected: GrpcConnectionSnapshot) -> Self {
Self(Arc::new(ConnectionGenerationGuardInner {
expected,
expected_boot_epoch: None,
outcome: AtomicU8::new(CONNECTION_GENERATION_PENDING),
}))
}
fn for_peer_capability(expected: GrpcConnectionSnapshot, boot_epoch: Uuid) -> Self {
Self(Arc::new(ConnectionGenerationGuardInner {
expected,
expected_boot_epoch: Some(boot_epoch),
outcome: AtomicU8::new(CONNECTION_GENERATION_PENDING),
}))
}
fn expected_peer_capability(&self) -> Option<(GrpcConnectionSnapshot, Uuid)> {
self.0.expected_boot_epoch.map(|boot_epoch| (self.0.expected, boot_epoch))
}
fn reject_before_dispatch(&self) {
let _ = self.0.outcome.compare_exchange(
CONNECTION_GENERATION_PENDING,
CONNECTION_GENERATION_REJECTED,
Ordering::AcqRel,
Ordering::Acquire,
);
}
fn validate_before_call(&self, current: Option<GrpcConnectionSnapshot>) -> Result<(), StaleConnectionGenerationError> {
if self.0.outcome.load(Ordering::Acquire) == CONNECTION_GENERATION_PENDING && current == Some(self.0.expected) {
return Ok(());
}
self.reject_before_dispatch();
Err(StaleConnectionGenerationError::new(self.0.expected, current))
}
fn begin_payload(&self, current: Option<GrpcConnectionSnapshot>) -> Result<(), tonic::Status> {
if current != Some(self.0.expected) {
self.reject_before_dispatch();
return Err(stale_connection_generation_status());
}
self.0
.outcome
.compare_exchange(
CONNECTION_GENERATION_PENDING,
CONNECTION_GENERATION_DISPATCHED,
Ordering::AcqRel,
Ordering::Acquire,
)
.map(|_| ())
.map_err(|_| stale_connection_generation_status())
}
fn rejected_before_dispatch(&self) -> bool {
self.0.outcome.load(Ordering::Acquire) == CONNECTION_GENERATION_REJECTED
}
}
fn stale_connection_generation_status() -> tonic::Status {
tonic::Status::aborted("internode connection generation changed before request dispatch")
}
#[derive(Debug)]
struct StaleConnectionGenerationError {
expected: GrpcConnectionSnapshot,
current: Option<GrpcConnectionSnapshot>,
status: tonic::Status,
}
impl StaleConnectionGenerationError {
fn new(expected: GrpcConnectionSnapshot, current: Option<GrpcConnectionSnapshot>) -> Self {
Self {
expected,
current,
status: stale_connection_generation_status(),
}
}
}
impl std::fmt::Display for StaleConnectionGenerationError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
write!(
formatter,
"internode connection changed before dispatch (expected {:?}, current {:?})",
self.expected, self.current
)
}
}
impl Error for StaleConnectionGenerationError {
fn source(&self) -> Option<&(dyn Error + 'static)> {
Some(&self.status)
}
}
pin_project_lite::pin_project! {
#[project = ConnectionGenerationFutureProj]
#[doc(hidden)]
pub enum ConnectionGenerationFuture<F, Response> {
Inner { #[pin] future: F },
Rejected { #[pin] future: std::future::Ready<Result<Response, StdError>> },
}
}
impl<F, Response, InnerError> Future for ConnectionGenerationFuture<F, Response>
where
F: Future<Output = Result<Response, InnerError>>,
InnerError: Into<StdError>,
{
type Output = Result<Response, StdError>;
fn poll(self: Pin<&mut Self>, context: &mut Context<'_>) -> Poll<Self::Output> {
match self.project() {
ConnectionGenerationFutureProj::Inner { future } => future.poll(context).map_err(Into::into),
ConnectionGenerationFutureProj::Rejected { future } => future.poll(context),
}
}
}
/// Rejects a stale encoded request before dispatching any body data to tonic's channel.
#[derive(Clone, Debug)]
pub struct ConnectionGenerationChannel<S> {
inner: S,
connection_state: Option<GrpcConnectionState>,
}
impl<S> ConnectionGenerationChannel<S> {
fn new(inner: S, connection_state: Option<GrpcConnectionState>) -> Self {
Self { inner, connection_state }
}
}
impl<S> Service<HttpRequest<tonic::body::Body>> for ConnectionGenerationChannel<S>
where
S: Service<HttpRequest<tonic::body::Body>>,
S::Error: Into<StdError>,
{
type Response = S::Response;
type Error = StdError;
type Future = ConnectionGenerationFuture<S::Future, S::Response>;
fn poll_ready(&mut self, context: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
self.inner.poll_ready(context).map_err(Into::into)
}
fn call(&mut self, mut request: HttpRequest<tonic::body::Body>) -> Self::Future {
let Some(guard) = request.extensions().get::<ConnectionGenerationGuard>().cloned() else {
return ConnectionGenerationFuture::Inner {
future: self.inner.call(request),
};
};
let current = self.connection_state.as_ref().and_then(GrpcConnectionState::snapshot);
if let Err(error) = guard.validate_before_call(current) {
return ConnectionGenerationFuture::Rejected {
future: std::future::ready(Err(Box::new(error))),
};
}
let body = std::mem::replace(request.body_mut(), tonic::body::Body::empty());
*request.body_mut() = tonic::body::Body::new(ConnectionGenerationBody {
inner: body,
guard,
connection_state: self.connection_state.clone(),
checked: false,
failed: false,
});
ConnectionGenerationFuture::Inner {
future: self.inner.call(request),
}
}
}
struct ConnectionGenerationBody {
inner: tonic::body::Body,
guard: ConnectionGenerationGuard,
connection_state: Option<GrpcConnectionState>,
checked: bool,
failed: bool,
}
impl http_body::Body for ConnectionGenerationBody {
type Data = bytes::Bytes;
type Error = tonic::Status;
fn poll_frame(
mut self: Pin<&mut Self>,
context: &mut Context<'_>,
) -> Poll<Option<Result<http_body::Frame<Self::Data>, Self::Error>>> {
if self.failed {
return Poll::Ready(None);
}
let frame = std::task::ready!(Pin::new(&mut self.inner).poll_frame(context));
if let Some(Ok(frame)) = &frame
&& frame.is_data()
&& !self.checked
{
let current = self.connection_state.as_ref().and_then(GrpcConnectionState::snapshot);
if let Err(error) = self.guard.begin_payload(current) {
self.failed = true;
return Poll::Ready(Some(Err(error)));
}
self.checked = true;
}
Poll::Ready(frame)
}
fn is_end_stream(&self) -> bool {
self.failed || self.inner.is_end_stream()
}
fn size_hint(&self) -> http_body::SizeHint {
self.inner.size_hint()
}
}
pub type AuthenticatedChannel = ReplayScopeChannel<BatchLockDeadlineChannel<InternodeChannel>>;
/// Gate lock-batch payloads after the transport's internal readiness queues.
/// Other RPC bodies pass through without an additional wrapper.
@@ -779,149 +515,6 @@ fn should_request_msgpack_capability(state: PeerReplayStateSnapshot) -> bool {
}
}
fn validate_peer_capability_guard(guard: &ConnectionGenerationGuard, state: PeerReplayStateSnapshot) {
let Some((expected_connection, expected_boot_epoch)) = guard.expected_peer_capability() else {
return;
};
let capability_is_current = state.connection == Some(expected_connection)
&& state.replay.boot_epoch == Some(expected_boot_epoch)
&& matches!(
state.msgpack_capability,
Some(PeerMsgpackCapability::Capable {
boot_epoch,
connection_generation,
}) if boot_epoch == expected_boot_epoch
&& connection_generation == expected_connection.generation
);
if !capability_is_current {
guard.reject_before_dispatch();
}
}
#[derive(Clone, Debug)]
enum MsgpackSendMode {
DualWrite,
ManualFleet,
PeerCapable { guard: ConnectionGenerationGuard },
}
/// One eligible RPC's immutable JSON/msgpack decision and optional pre-dispatch guard.
#[derive(Clone, Debug)]
pub(crate) struct MsgpackSendPolicy {
mode: MsgpackSendMode,
}
impl MsgpackSendPolicy {
#[cfg(test)]
pub(crate) fn for_test(omit_eligible_json: bool) -> Self {
Self {
mode: if omit_eligible_json {
MsgpackSendMode::ManualFleet
} else {
MsgpackSendMode::DualWrite
},
}
}
fn dual_write(reason: &'static str) -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics().record_msgpack_send_decision("dual_write", reason);
Self {
mode: MsgpackSendMode::DualWrite,
}
}
fn manual_fleet() -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics()
.record_msgpack_send_decision("msgpack_only", "manual_fleet");
Self {
mode: MsgpackSendMode::ManualFleet,
}
}
fn peer_capable(connection: GrpcConnectionSnapshot, boot_epoch: Uuid) -> Self {
rustfs_io_metrics::internode_metrics::global_internode_metrics()
.record_msgpack_send_decision("msgpack_only", "peer_capable");
Self {
mode: MsgpackSendMode::PeerCapable {
guard: ConnectionGenerationGuard::for_peer_capability(connection, boot_epoch),
},
}
}
pub(crate) fn omit_eligible_json(&self) -> bool {
!matches!(self.mode, MsgpackSendMode::DualWrite)
}
pub(crate) fn guard_request<T>(&self, request: &mut tonic::Request<T>) {
if let MsgpackSendMode::PeerCapable { guard } = &self.mode {
request.extensions_mut().insert(guard.clone());
}
}
/// Switch one peer-capability attempt to dual-write only after a proven local rejection.
pub(crate) fn retry_dual_write_after_local_rejection(&mut self) -> bool {
let rejected = matches!(
&self.mode,
MsgpackSendMode::PeerCapable { guard } if guard.rejected_before_dispatch()
);
if rejected {
*self = Self::dual_write("peer_revoked");
}
rejected
}
}
fn prepare_msgpack_send_policy(audience: Option<&str>, connection: Option<GrpcConnectionSnapshot>) -> MsgpackSendPolicy {
prepare_msgpack_send_policy_with_gates(
audience,
connection,
rustfs_protos::internode_rpc_msgpack_only(),
rustfs_protos::internode_rpc_msgpack_peer_capability_enabled(),
)
}
fn prepare_msgpack_send_policy_with_gates(
audience: Option<&str>,
connection: Option<GrpcConnectionSnapshot>,
manual_fleet: bool,
peer_capability: bool,
) -> MsgpackSendPolicy {
if manual_fleet {
return MsgpackSendPolicy::manual_fleet();
}
if !peer_capability {
return MsgpackSendPolicy::dual_write("not_requested");
}
let (Some(audience), Some(connection)) = (audience, connection) else {
return MsgpackSendPolicy::dual_write("peer_unknown");
};
let Ok(states) = PEER_REPLAY_STATES.lock() else {
return MsgpackSendPolicy::dual_write("peer_unknown");
};
let snapshot = states.snapshot(audience, Some(connection));
let (capable_epoch, revoked) = match (snapshot.replay.boot_epoch, snapshot.msgpack_capability) {
(
Some(boot_epoch),
Some(PeerMsgpackCapability::Capable {
boot_epoch: capability_epoch,
connection_generation,
}),
) if boot_epoch == capability_epoch && connection.generation == connection_generation => (Some(boot_epoch), false),
(_, Some(PeerMsgpackCapability::Revoked { connection_generation })) if connection.generation == connection_generation => {
(None, true)
}
_ => (None, false),
};
drop(states);
if let Some(boot_epoch) = capable_epoch {
MsgpackSendPolicy::peer_capable(connection, boot_epoch)
} else if revoked {
MsgpackSendPolicy::dual_write("peer_revoked")
} else {
MsgpackSendPolicy::dual_write("peer_unknown")
}
}
pub(crate) fn clear_peer_replay_state_for_addr(addr: &str) -> std::io::Result<()> {
let uri = addr
.parse::<Uri>()
@@ -1094,17 +687,7 @@ where
.get::<PeerReplayStateSnapshot>()
.copied()
.unwrap_or_default();
if let Some(guard) = request.extensions().get::<ConnectionGenerationGuard>() {
validate_peer_capability_guard(guard, sent_state);
}
let msgpack_capability_requested = authenticated && should_request_msgpack_capability(sent_state);
if msgpack_capability_requested
&& request.uri().path() == "/node_service.NodeService/Ping"
&& request.extensions().get::<ConnectionGenerationGuard>().is_none()
&& let Some(expected) = sent_state.connection
{
request.extensions_mut().insert(ConnectionGenerationGuard::new(expected));
}
if let (Some(audience), Some(challenge)) = (self.audience.as_deref(), challenge) {
// The challenge is independently HMAC-authenticated by the response proof. It is not
// part of v2 so old peers ignore it, while a new peer can safely advertise its epoch.
@@ -1428,106 +1011,6 @@ mod tests {
use tracing_opentelemetry::OpenTelemetrySpanExt;
use tracing_subscriber::{Registry, layer::SubscriberExt};
#[tokio::test]
async fn connection_generation_guard_rejects_only_local_predispatch_races() {
use http_body_util::BodyExt as _;
use std::sync::atomic::AtomicUsize;
#[derive(Clone)]
struct DrainBodyService {
calls: Arc<AtomicUsize>,
payloads: Arc<AtomicUsize>,
}
impl Service<HttpRequest<tonic::body::Body>> for DrainBodyService {
type Response = HttpResponse<tonic::body::Body>;
type Error = tonic::Status;
type Future = Pin<Box<dyn Future<Output = Result<Self::Response, Self::Error>> + Send>>;
fn poll_ready(&mut self, _context: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
Poll::Ready(Ok(()))
}
fn call(&mut self, request: HttpRequest<tonic::body::Body>) -> Self::Future {
self.calls.fetch_add(1, Ordering::SeqCst);
let payloads = Arc::clone(&self.payloads);
Box::pin(async move {
let mut body = request.into_body();
match body.frame().await {
Some(Ok(frame)) if frame.is_data() => {
payloads.fetch_add(1, Ordering::SeqCst);
Ok(HttpResponse::new(tonic::body::Body::empty()))
}
Some(Err(error)) => Err(error),
_ => Err(tonic::Status::internal("guard test body did not produce payload data")),
}
})
}
}
fn request(guard: ConnectionGenerationGuard) -> HttpRequest<tonic::body::Body> {
let mut request =
HttpRequest::new(tonic::body::Body::new(http_body_util::Full::new(bytes::Bytes::from_static(b"payload"))));
request.extensions_mut().insert(guard);
request
}
fn has_local_generation_source(status: &tonic::Status) -> bool {
let mut source = status.source();
while let Some(error) = source {
if error.downcast_ref::<StaleConnectionGenerationError>().is_some() {
return true;
}
source = error.source();
}
false
}
let state = GrpcConnectionState::new();
let first = state.record_connected();
let calls = Arc::new(AtomicUsize::new(0));
let payloads = Arc::new(AtomicUsize::new(0));
let service = DrainBodyService {
calls: Arc::clone(&calls),
payloads: Arc::clone(&payloads),
};
let mut channel = ConnectionGenerationChannel::new(service, Some(state.clone()));
let rejected_before_call = ConnectionGenerationGuard::new(first);
let second = state.record_connected();
let error = channel
.call(request(rejected_before_call.clone()))
.await
.expect_err("stale generation must fail before inner call");
let status = tonic::Status::from_error(error);
assert_eq!(status.code(), tonic::Code::Aborted);
assert!(has_local_generation_source(&status));
assert!(rejected_before_call.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 0);
assert_eq!(payloads.load(Ordering::SeqCst), 0);
assert!(!has_local_generation_source(&tonic::Status::aborted(status.message())));
let rejected_before_payload = ConnectionGenerationGuard::new(second);
let pending = channel.call(request(rejected_before_payload.clone()));
let third = state.record_connected();
let error = pending
.await
.expect_err("generation change before body poll must reject payload");
assert_eq!(tonic::Status::from_error(error).code(), tonic::Code::Aborted);
assert!(rejected_before_payload.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 1);
assert_eq!(payloads.load(Ordering::SeqCst), 0);
let dispatched = ConnectionGenerationGuard::new(third);
channel
.call(request(dispatched.clone()))
.await
.expect("matching generation dispatches payload");
assert!(!dispatched.rejected_before_dispatch());
assert_eq!(calls.load(Ordering::SeqCst), 2);
assert_eq!(payloads.load(Ordering::SeqCst), 1);
}
#[tokio::test]
async fn delete_recovery_channel_uses_its_scoped_deadline() {
use bytes::Bytes;
@@ -2573,68 +2056,6 @@ mod tests {
clear_peer_capability(audience);
}
#[test]
fn msgpack_send_policy_requires_exact_capability_and_retries_only_local_rejection() {
let audience = "msgpack-send-policy-test:9000";
let boot_epoch = Uuid::new_v4();
let connection = GrpcConnectionSnapshot {
channel_id: 21,
generation: 201,
};
clear_peer_capability(audience);
let unknown = PEER_REPLAY_STATES
.lock()
.expect("peer replay registry")
.snapshot(audience, Some(connection));
apply_peer_capabilities_response_for_connection(
audience.to_string(),
unknown,
Ok(authenticated_peer_response(boot_epoch, true)),
Some(Ok(authenticated_msgpack_response(boot_epoch, true))),
Some(connection),
);
let mut peer_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(connection), false, true);
assert!(peer_policy.omit_eligible_json());
let mut request = tonic::Request::new(());
peer_policy.guard_request(&mut request);
assert!(request.extensions().get::<ConnectionGenerationGuard>().is_some());
let MsgpackSendMode::PeerCapable { guard } = &peer_policy.mode else {
panic!("proved peer must use guarded msgpack-only policy");
};
let guard = guard.clone();
{
let mut states = PEER_REPLAY_STATES.lock().expect("peer replay registry");
states.set_msgpack_capability(
audience,
Some(connection),
PeerMsgpackCapability::Revoked {
connection_generation: connection.generation,
},
);
}
let revoked = PEER_REPLAY_STATES
.lock()
.expect("peer replay registry")
.snapshot(audience, Some(connection));
validate_peer_capability_guard(&guard, revoked);
assert!(peer_policy.retry_dual_write_after_local_rejection());
assert!(!peer_policy.omit_eligible_json());
assert!(!peer_policy.retry_dual_write_after_local_rejection());
let replacement = GrpcConnectionSnapshot {
channel_id: connection.channel_id,
generation: connection.generation + 1,
};
let unknown_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(replacement), false, true);
assert!(!unknown_policy.omit_eligible_json());
let manual_policy = prepare_msgpack_send_policy_with_gates(None, None, true, false);
assert!(manual_policy.omit_eligible_json());
let disabled_policy = prepare_msgpack_send_policy_with_gates(Some(audience), Some(connection), false, false);
assert!(!disabled_policy.omit_eligible_json());
clear_peer_capability(audience);
}
#[test]
fn clear_peer_replay_state_for_addr_removes_normalized_audience() {
let audience = "clear-peer-replay-state-test:9000";
+223 -345
View File
@@ -13,9 +13,8 @@
// limitations under the License.
use crate::cluster::rpc::client::{
AuthenticatedChannel, MsgpackSendPolicy, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
AuthenticatedChannel, TonicInterceptor, gen_tonic_signature_interceptor, is_network_like_disk_error,
node_service_delete_recovery_client, node_service_time_out_client, node_service_time_out_client_for_class,
node_service_time_out_client_for_class_with_msgpack_policy,
};
use crate::cluster::rpc::http_auth::{current_rpc_background_request, with_rpc_background_request};
use crate::cluster::rpc::internode_data_transport::{
@@ -2454,19 +2453,6 @@ impl RemoteDisk {
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn get_client_with_msgpack_policy(&self) -> Result<(NodeRpcClient, MsgpackSendPolicy)> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
node_service_time_out_client_for_class_with_msgpack_policy(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
ChannelClass::Control,
)
.await
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
/// Client for large `bytes`-carrying RPCs, including native pool metadata recovery inventories.
/// Routes onto the isolated bulk channel pool so large transfers cannot head-of-line block
/// lock/health RPCs (grpc-optimization P1). Falls back to the control channel when isolation
@@ -2507,19 +2493,6 @@ impl RemoteDisk {
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn get_bulk_client_with_msgpack_policy(&self) -> Result<(NodeRpcClient, MsgpackSendPolicy)> {
if let Some(err) = self.offline_bypass_error() {
return Err(err);
}
node_service_time_out_client_for_class_with_msgpack_policy(
&self.addr,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
ChannelClass::Bulk,
)
.await
.map_err(|err| Error::RemoteClientUnavailable(err.to_string()))
}
async fn disk_ref(&self) -> String {
(*self.id.lock().await)
.map(|id| id.to_string())
@@ -2562,8 +2535,8 @@ fn encode_file_info_versions_msgpack(value: &FileInfoVersions) -> Result<Vec<u8>
/// JSON compatibility string for a dual-encoded (`_bin` + text) request field. Returns an empty
/// string only when msgpack-only mode and its explicit fleet confirmation guard are both enabled;
/// otherwise the legacy JSON encoding is retained for old peers.
fn compat_json<T: Serialize>(value: &T, policy: &MsgpackSendPolicy) -> Result<String> {
if policy.omit_eligible_json() {
fn compat_json<T: Serialize>(value: &T) -> Result<String> {
if rustfs_protos::internode_rpc_msgpack_only() {
return Ok(String::new());
}
Ok(serde_json::to_string(value)?)
@@ -2837,12 +2810,26 @@ impl RemoteDisk {
|| async {
let prepare_started = rustfs_io_metrics::put_stage_timer();
let collect_prepare_timing = prepare_started.is_some();
let json_encode_started = collect_prepare_timing.then(Instant::now);
let file_info = compat_json(fi);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_ENCODE,
json_encode_started,
);
let file_info = file_info?;
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_BYTES,
file_info.len(),
);
}
let msgpack_encode_started = collect_prepare_timing.then(Instant::now);
let file_info_bin = Bytes::from(encode_file_info_msgpack(fi)?);
let file_info_bin = encode_file_info_msgpack(fi);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_MSGPACK_ENCODE,
msgpack_encode_started,
);
let file_info_bin = file_info_bin?;
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_MSGPACK_BYTES,
@@ -2850,57 +2837,35 @@ impl RemoteDisk {
);
}
let get_client_started = collect_prepare_timing.then(Instant::now);
let client = self.get_client_with_msgpack_policy().await;
let client = self.get_client().await;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_GET_CLIENT,
get_client_started,
);
let (mut client, mut policy) = client?;
let bucket_incarnation_id = crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
.map(|scope| Bytes::copy_from_slice(scope.incarnation.as_bytes()))
.unwrap_or_default();
let incarnation_bound = !bucket_incarnation_id.is_empty();
let mut client = client?;
let mut request = Request::new(RenameDataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk: self.endpoint.to_string(),
src_volume: src_volume.to_string(),
src_path: src_path.to_string(),
file_info,
dst_volume: dst_volume.to_string(),
dst_path: dst_path.to_string(),
file_info_bin: file_info_bin.into(),
scanner_publication_lease_token: scanner_publication_lease_token
.map(|token| token.as_bytes().to_vec().into())
.unwrap_or_default(),
});
let canonical_body_digest_started = collect_prepare_timing.then(Instant::now);
let incarnation_bound = !request.get_ref().bucket_incarnation_id.is_empty();
let quota_fenced = rustfs_utils::http::metadata_compat::contains_key_str(
&fi.metadata,
crate::disk::QUOTA_MUTATION_FENCE_METADATA_SUFFIX,
);
let scanner_publication_lease = scanner_publication_lease_token
.map(|token| Bytes::copy_from_slice(token.as_bytes()))
.unwrap_or_default();
let rpc_started = rustfs_io_metrics::put_stage_timer();
let mut retried_dual_write = false;
let response = loop {
let attempt_prepare_started = if retried_dual_write {
rustfs_io_metrics::put_stage_timer()
} else {
prepare_started
};
let json_encode_started = collect_prepare_timing.then(Instant::now);
let file_info = compat_json(fi, &policy)?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_ENCODE,
json_encode_started,
);
if collect_prepare_timing {
rustfs_io_metrics::record_put_object_stage_bytes(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_JSON_BYTES,
file_info.len(),
);
}
let mut request = Request::new(RenameDataRequest {
bucket_incarnation_id: bucket_incarnation_id.clone(),
disk: self.endpoint.to_string(),
src_volume: src_volume.to_string(),
src_path: src_path.to_string(),
file_info,
dst_volume: dst_volume.to_string(),
dst_path: dst_path.to_string(),
file_info_bin: file_info_bin.clone(),
scanner_publication_lease_token: scanner_publication_lease.clone(),
});
let canonical_body_digest_started = collect_prepare_timing.then(Instant::now);
let canonical_body_digest = (|| -> Result<()> {
let canonical_body = rustfs_protos::canonical_rename_data_request_body(request.get_ref());
policy.guard_request(&mut request);
if pool_meta_cas {
let body =
canonical_body.map_err(|_| Error::other("pool metadata CAS body length cannot be represented"))?;
@@ -2915,44 +2880,48 @@ impl RemoteDisk {
} else {
attach_mutation_body_digest(&mut request, canonical_body, "rename_data")?;
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_CANONICAL_BODY_DIGEST,
canonical_body_digest_started,
);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
attempt_prepare_started,
);
let result = rustfs_io_metrics::observe_put_stage_future(
async {
if multipart_rename {
client.rename_multipart_data(request).await
} else if pool_meta_cas {
client.rename_data_with_pool_meta_cas(request).await
} else if quota_fenced {
client.rename_data_with_quota_fence(request).await
} else if incarnation_bound {
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
}
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await;
match result {
Ok(response) => break response,
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
Ok(())
})();
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_CANONICAL_BODY_DIGEST,
canonical_body_digest_started,
);
canonical_body_digest?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_PREPARE,
prepare_started,
);
let rpc_started = rustfs_io_metrics::put_stage_timer();
let response = rustfs_io_metrics::observe_put_stage_future(
async {
if multipart_rename {
// Old peers must reject before an upload source can be changed.
client.rename_multipart_data(request).await
} else if pool_meta_cas {
client.rename_data_with_pool_meta_cas(request).await
} else if quota_fenced {
// The method itself proves quota-token enforcement; an old peer must fail closed.
client.rename_data_with_quota_fence(request).await
} else if incarnation_bound {
// Older peers return Unimplemented before mutation; never downgrade.
client.rename_data_at_incarnation(request).await
} else {
client.rename_data(request).await
}
Err(status) if (multipart_rename || pool_meta_cas) && status.code() == Code::Unimplemented => {
return Err(DiskError::UnsupportedDisk);
}
Err(status) => return Err(status.into()),
},
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_POLL_PENDING_COUNT,
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_REMOTE_CLIENT_RPC_AWAIT_FIRST_PENDING_TO_READY,
)
.await
.map_err(|status| {
if (multipart_rename || pool_meta_cas) && status.code() == Code::Unimplemented {
DiskError::UnsupportedDisk
} else {
status.into()
}
};
})?;
let response_process_started = rustfs_io_metrics::put_stage_timer();
let response_headers_started = rustfs_io_metrics::put_stage_timer();
let _ = response.metadata().len();
@@ -3787,45 +3756,34 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let file_info_bin = Bytes::from(encode_file_info_msgpack(&fi)?);
let file_info = compat_json(&fi)?;
let file_info_bin = encode_file_info_msgpack(&fi)?;
self.execute_with_timeout_for_op(
"write_metadata",
move || async move {
let disk = self.disk_ref().await;
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let file_info = compat_json(&fi, &policy)?;
let mut request = Request::new(WriteMetadataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope(volume)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
file_info,
file_info_bin: file_info_bin.clone(),
});
let canonical_body = rustfs_protos::canonical_write_metadata_request_body(request.get_ref());
policy.guard_request(&mut request);
let result = if request.get_ref().bucket_incarnation_id.is_empty() {
attach_mutation_body_digest(&mut request, canonical_body, "write_metadata")?;
client.write_metadata(request).await
} else {
let body =
canonical_body.map_err(|_| Error::other("write metadata request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?;
client.write_metadata_at_incarnation(request).await
};
match result {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
let mut client = self.get_client().await?;
let mut request = Request::new(WriteMetadataRequest {
bucket_incarnation_id: crate::store::bucket_heal_scope(volume)
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
.unwrap_or_default(),
disk,
volume: volume.to_string(),
path: path.to_string(),
file_info,
file_info_bin: file_info_bin.into(),
});
let canonical_body = rustfs_protos::canonical_write_metadata_request_body(request.get_ref());
let response = if request.get_ref().bucket_incarnation_id.is_empty() {
attach_mutation_body_digest(&mut request, canonical_body, "write_metadata")?;
client.write_metadata(request).await?
} else {
let body = canonical_body.map_err(|_| Error::other("write metadata request length cannot be represented"))?;
crate::cluster::rpc::set_tonic_canonical_body_digest(&mut request, &body).map_err(Error::other)?;
client.write_metadata_at_incarnation(request).await?
}
.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -3879,38 +3837,29 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let file_info_bin = Bytes::from(encode_file_info_msgpack(&fi)?);
let opts_bin = Bytes::from(encode_msgpack(opts)?);
let file_info = compat_json(&fi)?;
let opts_str = compat_json(&opts)?;
let file_info_bin = encode_file_info_msgpack(&fi)?;
let opts_bin = encode_msgpack(opts)?;
self.execute_with_timeout_for_op(
"update_metadata",
move || async move {
let disk = self.disk_ref().await;
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let file_info = compat_json(&fi, &policy)?;
let opts_str = compat_json(opts, &policy)?;
let mut request = Request::new(UpdateMetadataRequest {
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
file_info,
opts: opts_str,
file_info_bin: file_info_bin.clone(),
opts_bin: opts_bin.clone(),
});
let canonical_body = rustfs_protos::canonical_update_metadata_request_body(request.get_ref());
policy.guard_request(&mut request);
attach_mutation_body_digest(&mut request, canonical_body, "update_metadata")?;
match client.update_metadata(request).await {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
let mut client = self.get_client().await?;
let mut request = Request::new(UpdateMetadataRequest {
disk,
volume: volume.to_string(),
path: path.to_string(),
file_info,
opts: opts_str,
file_info_bin: file_info_bin.into(),
opts_bin: opts_bin.into(),
});
let canonical_body = rustfs_protos::canonical_update_metadata_request_body(request.get_ref());
attach_mutation_body_digest(&mut request, canonical_body, "update_metadata")?;
let response = client.update_metadata(request).await?.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -3946,8 +3895,9 @@ impl DiskAPI for RemoteDisk {
);
let read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let encode_started = read_version_stage_timer(read_version_attribution_enabled);
let opts_bin = Bytes::from(encode_msgpack(opts)?);
let encoded_opts = compat_json(opts).and_then(|opts_str| encode_msgpack(opts).map(|opts_bin| (opts_str, opts_bin)));
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_REQUEST_ENCODE, encode_started);
let (opts_str, opts_bin) = encoded_opts?;
// Idempotent version read: eligible for the bounded transient-network retry so a single
// reset-by-peer during the read-after-write window does not erode the metadata read
@@ -3956,51 +3906,43 @@ impl DiskAPI for RemoteDisk {
self.execute_read_with_retry(
"read_version",
|| async {
let opts_str = opts_str.clone();
let opts_bin = opts_bin.clone();
let disk = self.disk_ref().await;
let (mut client, mut policy) = self.get_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let opts_str = compat_json(opts, &policy)?;
let request_payload_bytes = read_version_attribution_enabled.then(|| {
disk.len()
.saturating_add(volume.len())
.saturating_add(path.len())
.saturating_add(version_id.len())
.saturating_add(opts_str.len())
.saturating_add(opts_bin.len())
});
let mut request = Request::new(ReadVersionRequest {
disk: disk.clone(),
volume: volume.to_string(),
path: path.to_string(),
version_id: version_id.to_string(),
opts: opts_str,
opts_bin: opts_bin.clone(),
});
policy.guard_request(&mut request);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(
request_payload_bytes,
);
let mut client = self.get_client().await?;
let request_payload_bytes = read_version_attribution_enabled.then(|| {
disk.len()
.saturating_add(volume.len())
.saturating_add(path.len())
.saturating_add(version_id.len())
.saturating_add(opts_str.len())
.saturating_add(opts_bin.len())
});
let request = Request::new(ReadVersionRequest {
disk,
volume: volume.to_string(),
path: path.to_string(),
version_id: version_id.to_string(),
opts: opts_str,
opts_bin: opts_bin.into(),
});
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_request(opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_sent_bytes(request_payload_bytes);
}
let rpc_started = read_version_stage_timer(read_version_attribution_enabled);
let response = match client.read_version(request).await {
Ok(response) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
response.into_inner()
}
let rpc_started = read_version_stage_timer(read_version_attribution_enabled);
match client.read_version(request).await {
Ok(response) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
break response.into_inner();
}
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
retried_dual_write = true;
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error(
crate::cluster::rpc::runtime_sources::ReadVersionErrorStage::RpcStatus,
);
return Err(err.into());
}
Err(err) => {
record_read_version_stage(INTERNODE_STAGE_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_read_version_error(
crate::cluster::rpc::runtime_sources::ReadVersionErrorStage::RpcStatus,
);
return Err(err.into());
}
};
@@ -4062,13 +4004,14 @@ impl DiskAPI for RemoteDisk {
state = "started",
"Remote disk RPC started"
);
let req = Arc::new(req);
let batch_read_version_attribution_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let encode_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let batch_read_version_req_bin = Bytes::from(encode_msgpack(req.as_ref())?);
let batch_read_version_req = compat_json(&req)?;
let batch_read_version_req_bin = encode_msgpack(&req)?;
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_REQUEST_ENCODE, encode_started);
let batch_rpc_req = Arc::clone(&req);
let request_opts = batch_rpc_req.opts;
let request_payload_bytes = batch_read_version_attribution_enabled
.then(|| batch_read_version_request_payload_len(&req, &batch_read_version_req, &batch_read_version_req_bin));
let request_opts = req.opts;
let bulk_body = self.acquire_bulk_body().await?;
let batch_result = self
.execute_with_timeout_for_op(
@@ -4076,61 +4019,31 @@ impl DiskAPI for RemoteDisk {
move || async move {
let disk = self.disk_ref().await;
let disk_len = disk.len();
let (mut client, mut policy) = self.get_bulk_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let batch_read_version_req = compat_json(batch_rpc_req.as_ref(), &policy)?;
let request_payload_bytes = batch_read_version_attribution_enabled.then(|| {
batch_read_version_request_payload_len(
batch_rpc_req.as_ref(),
&batch_read_version_req,
&batch_read_version_req_bin,
)
});
let mut request = Request::new(BatchReadVersionRequest {
disk: disk.clone(),
batch_read_version_req,
batch_read_version_req_bin: batch_read_version_req_bin.clone(),
});
policy.guard_request(&mut request);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
request_payload_bytes.saturating_add(disk_len),
);
}
let rpc_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let response_result =
measure_client_rpc(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, client.batch_read_version(request))
.await;
match response_result {
Ok(response) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
break response.into_inner();
}
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
retried_dual_write = true;
}
Err(status) if status.code() == Code::Unimplemented => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
if mode.should_fallback_on_unimplemented() {
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_FALLBACK_UNIMPLEMENTED);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "fallback_unimplemented",
"Remote disk BatchReadVersion unsupported; falling back to unary read_version"
);
return Ok(None);
}
let mut client = self.get_bulk_client().await?;
let request = Request::new(BatchReadVersionRequest {
disk,
batch_read_version_req,
batch_read_version_req_bin: batch_read_version_req_bin.into(),
});
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_UNSUPPORTED_NO_FALLBACK);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_request(&request_opts);
if let Some(request_payload_bytes) = request_payload_bytes {
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_sent_bytes(
request_payload_bytes.saturating_add(disk_len),
);
}
let rpc_started = read_version_stage_timer(batch_read_version_attribution_enabled);
let response_result =
measure_client_rpc(INTERNODE_OPERATION_GRPC_BATCH_READ_VERSION, client.batch_read_version(request)).await;
let response = match response_result {
Ok(response) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
response.into_inner()
}
Err(status) if status.code() == Code::Unimplemented => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
if mode.should_fallback_on_unimplemented() {
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_FALLBACK_UNIMPLEMENTED);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
@@ -4138,16 +4051,30 @@ impl DiskAPI for RemoteDisk {
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "unsupported_no_fallback",
"Remote disk BatchReadVersion unsupported and explicit batch RPC mode forbids fallback"
state = "fallback_unimplemented",
"Remote disk BatchReadVersion unsupported; falling back to unary read_version"
);
return Err(Error::from(status));
}
Err(status) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
return Err(Error::from(status));
return Ok(None);
}
record_batch_read_version_gate_decision(mode, BATCH_READ_VERSION_GATE_UNSUPPORTED_NO_FALLBACK);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
batch_metadata_rpc_mode = mode.as_str(),
op = "batch_read_version",
state = "unsupported_no_fallback",
"Remote disk BatchReadVersion unsupported and explicit batch RPC mode forbids fallback"
);
return Err(Error::from(status));
}
Err(status) => {
record_batch_read_version_stage(INTERNODE_STAGE_BATCH_READ_VERSION_RPC_ROUNDTRIP, rpc_started);
crate::cluster::rpc::runtime_sources::record_remote_disk_grpc_batch_read_version_error();
return Err(Error::from(status));
}
};
@@ -4183,7 +4110,6 @@ impl DiskAPI for RemoteDisk {
// instead of racing the whole batch against one drive timeout.
None => {
drop(bulk_body);
let req = Arc::try_unwrap(req).unwrap_or_else(|req| (*req).clone());
batch_read_version_one_by_one(self, req).await
}
}
@@ -5130,27 +5056,18 @@ impl DiskAPI for RemoteDisk {
let _bulk_body = self.acquire_bulk_body().await?;
self.execute_with_timeout(
|| async {
let read_multiple_req_bin = Bytes::from(encode_msgpack(&req)?);
let read_multiple_req = compat_json(&req)?;
let read_multiple_req_bin = encode_msgpack(&req)?;
let disk = self.disk_ref().await;
let (mut client, mut policy) = self.get_bulk_client_with_msgpack_policy().await?;
let mut retried_dual_write = false;
let response = loop {
let read_multiple_req = compat_json(&req, &policy)?;
let mut request = Request::new(ReadMultipleRequest {
disk: disk.clone(),
read_multiple_req,
read_multiple_req_bin: read_multiple_req_bin.clone(),
});
policy.guard_request(&mut request);
let result = measure_client_rpc(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, client.read_multiple(request)).await;
match result {
Ok(response) => break response.into_inner(),
Err(_) if !retried_dual_write && policy.retry_dual_write_after_local_rejection() => {
retried_dual_write = true;
}
Err(status) => return Err(status.into()),
}
};
let mut client = self.get_bulk_client().await?;
let request = Request::new(ReadMultipleRequest {
disk,
read_multiple_req,
read_multiple_req_bin: read_multiple_req_bin.into(),
});
let response = measure_client_rpc(INTERNODE_OPERATION_GRPC_READ_MULTIPLE, client.read_multiple(request)).await;
let response = response?.into_inner();
if !response.success {
return Err(response.error.unwrap_or_default().into());
@@ -6337,40 +6254,6 @@ mod tests {
}
}
#[test]
fn msgpack_eligible_requests_bind_one_policy_and_bounded_local_retry() {
let source = rustfs_protos::compat_manifest::production_source(include_str!("remote_disk.rs"), "remote_disk.rs");
for function in [
"rename_data_borrowed_with_fence",
"write_metadata",
"update_metadata",
"read_version",
"batch_read_version",
"read_multiple",
] {
let start = source
.find(&format!("async fn {function}("))
.unwrap_or_else(|| panic!("missing {function}"));
let tail = &source[start..];
let next_instrument = ["\n #[tracing::", "instrument"].concat();
let end = tail[1..].find(&next_instrument).map_or(tail.len(), |offset| offset + 1);
let body = &tail[..end];
assert!(body.contains("with_msgpack_policy().await"), "{function} must capture one RPC policy");
assert!(
body.contains("policy.guard_request(&mut request)"),
"{function} must bind the generation guard"
);
assert!(
body.contains("policy.retry_dual_write_after_local_rejection()"),
"{function} must allow only the bounded local dual-write retry"
);
assert!(
body.contains("let mut retried_dual_write = false;"),
"{function} must bound retry to one attempt"
);
}
}
#[test]
fn delete_options_named_wire_preserves_rollback_and_legacy_marker_fields() {
#[derive(Debug, serde::Serialize, serde::Deserialize, PartialEq)]
@@ -7279,8 +7162,7 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
let json = compat_json(&resp).expect("compat_json should encode");
assert!(!json.is_empty());
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
},
@@ -7305,8 +7187,7 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
let json = compat_json(&resp).expect("compat_json should encode");
assert!(!json.is_empty(), "old JSON peers must remain compatible without fleet confirmation");
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
@@ -7323,8 +7204,7 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
let json = compat_json(&resp).expect("compat_json should encode");
assert!(json.is_empty(), "msgpack-only may empty JSON only after explicit fleet confirmation");
},
@@ -7340,8 +7220,7 @@ mod tests {
],
|| {
let resp = sample_read_multiple_resp("file", b"data");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
let json = compat_json(&resp).expect("compat_json should encode");
assert!(json.is_empty(), "both gates should enter msgpack-only send mode");
},
@@ -7359,8 +7238,7 @@ mod tests {
] {
with_internode_msgpack_env(vars, || {
let resp = sample_read_multiple_resp("file", b"data");
let policy = MsgpackSendPolicy::for_test(rustfs_protos::internode_rpc_msgpack_only());
let json = compat_json(&resp, &policy).expect("compat_json should encode");
let json = compat_json(&resp).expect("compat_json should encode");
assert!(!json.is_empty(), "removing either gate should restore old-peer JSON compatibility");
assert_eq!(json, serde_json::to_string(&resp).expect("json should encode"));
+119 -21
View File
@@ -1463,22 +1463,36 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
let observed = load_observed_data_usage_snapshot(store.clone()).await.ok().flatten();
let (selected, selected_is_current_format) =
select_admin_data_usage_snapshot(authoritative, source.is_authoritative(), observed);
let mut response = normalize_loaded_data_usage(selected, selected_is_current_format).await.0;
Ok(normalize_loaded_data_usage(selected, selected_is_current_format).await.0)
}
async fn project_admin_data_usage_for_current_buckets(
store: Arc<ECStore>,
mut response: DataUsageInfo,
with_memory_overlay: bool,
) -> DataUsageInfo {
use futures::StreamExt as _;
let snapshot_measured = response.is_complete_bucket_usage_snapshot() || response.is_valid_partial_snapshot();
let overlay_authoritative = with_memory_overlay && !runtime_sources::setup_is_dist_erasure().await;
let deadline = tokio::time::Instant::now() + ADMIN_USAGE_INCARNATION_TIMEOUT;
let mut incarnations = HashMap::new();
{
// Display-only identity reads must not serialize every bucket behind a
// slow metadata lock while holding the shared admin refresh permit.
// The common deadline bounds their total added latency; dropping this
// stream cancels all unfinished reads without spawning detached work.
// Own the names and store across awaits so this future remains Send
// when composed into spawned metrics collectors.
let buckets = response
.usage_snapshot_bucket_observations
// Identity reads happen after releasing snapshot and refresh locks,
// including on cache hits. Metadata transaction reads linearize each
// bucket against creation/deletion without relying on peer notices.
// The common deadline cancels unfinished work and bounds added latency.
let mut buckets = response
.buckets_usage
.keys()
.chain(response.usage_snapshot_bucket_observations.keys())
.cloned()
.collect::<Vec<_>>();
.collect::<HashSet<_>>();
if overlay_authoritative {
// Copy only names while holding the memory lock, then release it
// before storage I/O. A later new entry has no identity proof and
// will remain unknown in this response.
buckets.extend(memory_cache().read().await.keys().cloned());
}
let mut identities = futures::stream::iter(buckets)
.map(move |bucket| {
let store = store.clone();
@@ -1492,7 +1506,6 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
let Ok(Some((bucket, identity))) = tokio::time::timeout_at(deadline, identities.next()).await else {
break;
};
// Unavailable, timed-out or recreated buckets retain canonical values.
if let Ok(incarnation) = identity
&& !incarnation.is_nil()
{
@@ -1501,7 +1514,20 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
}
}
response.project_bucket_observations_for_admin(&incarnations);
Ok(response)
if with_memory_overlay
&& apply_bucket_usage_memory_overlay_if_authoritative(&mut response, overlay_authoritative).await
&& snapshot_measured
{
response.usage_snapshot_complete = false;
response.usage_snapshot_partial = true;
response.usage_snapshot_converged = Some(false);
response.usage_snapshot_admin_projection = true;
response.replication_info.clear();
response.tier_stats = None;
response.unknown_tier_stats = None;
}
response.retain_current_bucket_usage_for_admin(&incarnations);
response
}
fn discard_incomplete_bucket_usage(data_usage_info: &mut DataUsageInfo) {
@@ -1690,11 +1716,7 @@ async fn project_data_usage_from_backend_cached<T>(
}
}
/// Load the freshest structurally complete snapshot for authenticated admin
/// observability. A scan raced by namespace activity may be selected here, but
/// never by [`load_data_usage_from_backend_cached`], which remains the
/// converged source for quota admission.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
loop {
@@ -1743,6 +1765,38 @@ pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> R
}
}
/// Load cached scanner statistics for authenticated admin observability.
///
/// Each response verifies bucket identities against current storage metadata,
/// including cache hits. Missing, failed, or timed-out identity reads leave that
/// bucket's usage absent (unknown). Legacy values without an incarnation stay
/// unknown until a scanner snapshot certifies their identity. Per-bucket reads
/// linearize independently; this is not an atomic namespace snapshot.
/// Identity reads have a shared deadline and do not outlive cancellation. The
/// unchanged cached baseline remains independent of quota and persistence.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, false).await)
}
/// Load the admin view with the existing authoritative single-process overlay.
///
/// The overlay is applied before current-identity projection so it cannot
/// reintroduce statistics from a retired or unverified bucket generation.
/// Distributed nodes continue to use scanner statistics without local totals.
/// Cancellation, unknown values, and per-bucket read semantics are the same as
/// [`load_admin_data_usage_from_backend_cached`]. No overlay is persisted.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_with_memory_overlay(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, true).await)
}
/// Invalidate the process-local persisted usage snapshot cache after a durable
/// scanner write.
pub async fn invalidate_data_usage_snapshot_cache() {
@@ -2805,16 +2859,17 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
*cache = next_cache;
}
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) {
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) -> bool {
if !authoritative {
return;
return false;
}
let cache = memory_cache().read().await;
if cache.is_empty() {
return;
return false;
}
let was_complete = data_usage_info.usage_snapshot_complete;
let persisted_update = data_usage_info.last_update;
let mut changed = false;
let mut added_bucket = false;
@@ -2851,6 +2906,7 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
data_usage_info.calculate_totals();
}
changed || was_complete != data_usage_info.usage_snapshot_complete
}
pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
@@ -3744,6 +3800,23 @@ mod tests {
*cache_updating().write().await = false;
}
async fn bind_admin_usage_test_identities(store: &Arc<ECStore>, snapshot: &mut DataUsageInfo) {
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
for (bucket, usage) in &mut snapshot.buckets_usage {
store
.make_bucket(bucket, &crate::storage_api_contracts::bucket::MakeBucketOptions::default())
.await
.expect("create a real bucket for current admin identity validation");
usage.bucket_incarnation = Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("read the storage-owned fixture identity")
.to_string(),
);
}
}
fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
let mut info = DataUsageInfo {
last_update: Some(last_update),
@@ -4279,7 +4352,11 @@ mod tests {
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
let (mut authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
bind_admin_usage_test_identities(&store, &mut authoritative).await;
for (bucket, usage) in &mut observed.buckets_usage {
usage.bucket_incarnation = authoritative.buckets_usage[bucket].bucket_incarnation.clone();
}
let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
@@ -4328,6 +4405,11 @@ mod tests {
reconciled.scanner_epoch = Some(4);
reconciled.scanner_cycle = Some(12);
reconciled.usage_snapshot_converged = Some(true);
reconciled
.buckets_usage
.get_mut("large-bucket")
.expect("reconciled bucket")
.bucket_incarnation = authoritative.buckets_usage["large-bucket"].bucket_incarnation.clone();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4849,6 +4931,12 @@ mod tests {
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut current).await;
backup
.buckets_usage
.get_mut("existing")
.expect("backup bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
@@ -4871,6 +4959,11 @@ mod tests {
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
after_delete
.buckets_usage
.get_mut("existing")
.expect("post-delete bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
@@ -4932,6 +5025,7 @@ mod tests {
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut authoritative).await;
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4946,6 +5040,10 @@ mod tests {
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info.buckets_usage
.get_mut("bucket")
.expect("observed bucket")
.bucket_incarnation = authoritative.buckets_usage["bucket"].bucket_incarnation.clone();
info
};
let first = observation(7, 11, 200, 11, 1_100);
@@ -221,7 +221,7 @@ async fn companion_admin_cache_isolated_from_quota_memory_and_persistence() {
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buckets() {
async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_buckets() {
use crate::bucket::metadata_sys::{
acquire_metadata_transaction_guard, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
@@ -280,8 +280,11 @@ async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buc
.await
.expect("optional projection must return before the blocked metadata lock's long timeout")
.expect("canonical admin response stays available");
assert_eq!(response.buckets_usage[blocked].size, 100);
assert_eq!(response.buckets_usage[blocked].usage_value_state, Some(BucketUsageValueState::Carried));
assert!(
!response.buckets_usage.contains_key(blocked),
"an unavailable current identity cannot bind the canonical value"
);
assert!(!response.bucket_sizes.contains_key(blocked));
assert_eq!(response.buckets_usage[healthy].size, 250);
assert_eq!(response.buckets_usage[healthy].usage_value_state, Some(BucketUsageValueState::Observed));
assert!(response.usage_snapshot_admin_projection);
@@ -293,13 +296,12 @@ async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buc
.is_ok()
);
drop(blocked_guard);
invalidate_data_usage_snapshot_cache().await;
let recovered = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh after lock recovery");
assert_eq!(
recovered.buckets_usage[blocked].size, 250,
"unavailable identity can recover on the next refresh"
"unavailable identity can recover on a cache hit"
);
assert_eq!(recovered.buckets_usage[healthy].size, 250);
let quota = load_data_usage_from_backend_cached(store)
@@ -308,3 +310,201 @@ async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buc
assert_eq!(quota.buckets_usage[blocked].size, 100);
assert_eq!(quota.buckets_usage[healthy].size, 100);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn cached_canonical_admin_usage_rechecks_current_identity_without_peer_invalidation() {
use crate::bucket::metadata::{load_bucket_incarnation, save_bucket_incarnation};
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-cache";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
canonical.usage_snapshot_pending_buckets.clear();
canonical.usage_snapshot_pending_bucket_count = 0;
canonical
.buckets_usage
.get_mut(bucket)
.expect("canonical bucket")
.objects_count = 107;
canonical.calculate_totals();
let original_bytes = serde_json::to_vec(&canonical).expect("encode canonical old bucket");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), original_bytes.clone())
.await
.expect("persist canonical root");
invalidate_data_usage_snapshot_cache().await;
let first = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("prime raw admin cache");
assert_eq!(first.buckets_usage[bucket].objects_count, 107);
assert!(first.is_complete_bucket_usage_snapshot());
let cache_generation = admin_data_usage_snapshot_generation();
let new_identity = uuid::Uuid::new_v4();
// Simulate the storage-owned identity changing on another node while this
// process has not received its invalidation notice. Do not clear the cache.
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish changed disk identity");
assert_eq!(
load_bucket_incarnation(store.clone(), bucket)
.await
.expect("read new sidecar"),
Some(new_identity)
);
let second = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("read raw cache with fresh identity");
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
assert!(
!second.buckets_usage.contains_key(bucket),
"same name cannot retain the predecessor's 107"
);
assert!(!second.bucket_sizes.contains_key(bucket));
assert_eq!(second.usage_snapshot_converged, Some(false));
assert!(second.usage_snapshot_admin_projection);
assert!(reject_admin_usage_projection(&second).is_err());
assert!(certified_bucket_usage(&second, bucket, new_identity).is_none());
assert!(parse_usage_snapshot(&serde_json::to_vec(&second).expect("encode display response")).is_err());
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("independent quota baseline");
assert_eq!(quota.buckets_usage[bucket].objects_count, 107);
assert_eq!(
read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("unchanged persisted root"),
original_bytes
);
// Rebinding a cached raw baseline is transient: recovery does not need to
// reload or permanently blank a healthy bucket's statistics.
save_bucket_incarnation(store.clone(), bucket, old_identity)
.await
.expect("restore fixture identity");
let recovered = load_admin_data_usage_from_backend_cached(store)
.await
.expect("recover on another cache hit");
assert_eq!(recovered.buckets_usage[bucket].objects_count, 107);
assert!(recovered.is_complete_bucket_usage_snapshot());
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_cannot_reintroduce_a_predecessor_identity() {
use crate::bucket::metadata::save_bucket_incarnation;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-overlay";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode canonical root"),
)
.await
.expect("persist overlay fixture");
invalidate_data_usage_snapshot_cache().await;
let cached_usage = canonical.buckets_usage[bucket].clone();
replace_bucket_usage_memory_from_authoritative(bucket, cached_usage, SystemTime::now()).await;
let new_identity = uuid::Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish a replacement identity");
let response = load_admin_data_usage_with_memory_overlay(store.clone())
.await
.expect("project after memory overlay");
assert!(!response.buckets_usage.contains_key(bucket));
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(response.usage_snapshot_admin_projection);
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_overlay_flags_only_change_keeps_identity_bound_cold_usage() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(
!runtime_sources::setup_is_dist_erasure().await,
"exercise the production single-process overlay"
);
let cold = "usage-overlay-cold";
let unverified = "usage-overlay-unverified";
for bucket in [cold, unverified] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay fixture bucket");
}
let identity = store
.bucket_incarnation_id_from_disk(cold)
.await
.expect("cold bucket identity");
let mut canonical = companion_snapshot(cold, identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode cold canonical value"),
)
.await
.expect("persist cold baseline");
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
memory_cache().write().await.insert(
unverified.to_string(),
cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false),
);
record_bucket_object_write_memory(unverified, None, 42).await;
{
let cache = memory_cache().read().await;
assert!(cache[unverified].dirty);
assert!(!cache[unverified].authoritative);
}
let response = load_admin_data_usage_with_memory_overlay(store)
.await
.expect("identity-filtered flags-only overlay");
assert_eq!(response.buckets_usage[cold].size, 100);
assert_eq!(response.objects_total_size, 100);
assert_eq!(response.bucket_sizes[cold], 100);
assert!(!response.buckets_usage.contains_key(unverified));
assert!(response.usage_snapshot_admin_projection);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
@@ -1033,16 +1033,26 @@ mod tests {
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket("bucket", &MakeBucketOptions::default())
.await
.expect("create a real observed bucket");
let incarnation = store
.bucket_incarnation_id_from_disk("bucket")
.await
.expect("read current storage-owned incarnation");
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
@@ -1066,6 +1076,7 @@ mod tests {
versions_count: 4,
delete_markers_count: 1,
size: 6,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
},
)]
@@ -1117,6 +1128,46 @@ mod tests {
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn identity_filtered_usage_remains_unknown_in_server_info() {
let mut info = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
buckets_count: 1,
objects_total_count: 107,
objects_total_size: 1024,
buckets_usage: [(
"bucket".to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 107,
size: 1024,
bucket_incarnation: Some("11111111-1111-4111-8111-111111111111".to_string()),
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
..Default::default()
};
let current =
std::collections::HashMap::from([("bucket".to_string(), "22222222-2222-4222-8222-222222222222".to_string())]);
assert_eq!(info.retain_current_bucket_usage_for_admin(&current), 1);
assert!(info.buckets_usage.is_empty());
assert!(info.usage_snapshot_admin_projection);
assert_eq!(info.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
+323 -2
View File
@@ -11083,6 +11083,104 @@ impl LocalDisk {
return Ok(());
}
if !recursive
&& staging_cleanup.is_none()
&& delete_path.file_name().and_then(|n| n.to_str()) == Some(STORAGE_FORMAT_FILE_BACKUP)
&& self.multipart_delete_check(delete_path, false)?.is_none()
{
let target_path = delete_path.clone();
let root_base = base_path.to_path_buf();
let prune_op = move |is_cancelled: &dyn Fn() -> bool, check_cancel: bool| -> Result<()> {
if check_cancel && is_cancelled() {
return Err(DiskError::Io(std::io::Error::other("blocking namespace operation cancelled")));
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(os::prepared_publication_test_hooks::Stage::Remove, &target_path);
#[cfg(not(target_os = "linux"))]
let start_as_dir = target_path.is_dir();
#[cfg(target_os = "linux")]
let start_as_dir = false;
let mut current = if start_as_dir {
Some(target_path.as_path())
} else {
match std::fs::remove_file(&target_path) {
Ok(()) => target_path.parent(),
Err(err) if err.kind() == ErrorKind::NotFound => target_path.parent(),
Err(err)
if err.kind() == ErrorKind::IsADirectory
|| (cfg!(not(target_os = "linux"))
&& err.kind() == ErrorKind::PermissionDenied
&& target_path.is_dir()) =>
{
Some(target_path.as_path())
}
Err(err) => {
warn!(
event = EVENT_DISK_LOCAL_DELETE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
path = ?target_path,
operation = "remove_file",
error = ?err,
"Disk local delete failed"
);
return Err(Error::other(FileAccessDeniedWithContext {
path: target_path.clone(),
source: err,
}));
}
}
};
let first_dir = current;
while let Some(dir_path) = current.filter(|parent| *parent != root_base && parent.starts_with(&root_base)) {
if Some(dir_path) != first_dir && std::fs::symlink_metadata(dir_path.join(STORAGE_FORMAT_FILE)).is_ok() {
return Ok(());
}
if dir_path != target_path.as_path() {
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
os::prepared_publication_test_hooks::run(
os::prepared_publication_test_hooks::Stage::Remove,
dir_path,
);
}
match std::fs::remove_dir(dir_path) {
Ok(()) => current = dir_path.parent(),
Err(err) if is_dir_not_empty_error(&err) => return Ok(()),
Err(err) if is_benign_object_rmdir_error(&err) => current = dir_path.parent(),
Err(err) => {
warn!(
event = EVENT_DISK_LOCAL_DELETE_FAILED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
path = ?dir_path,
operation = "remove_dir",
error_kind = %err.kind(),
"Disk local delete failed"
);
return Err(Error::other(FileAccessDeniedWithContext {
path: dir_path.to_path_buf(),
source: err,
}));
}
}
}
Ok(())
};
return match namespace_owner {
Some(owner) => {
let lease = os::acquire_namespace_mutation_lease_with_owner(delete_path, Some(owner)).await;
os::run_blocking_namespace_operation(lease, move || Ok(prune_op(&|| false, false)))
.await
.map_err(to_file_error)?
}
None => self
.blocking_queue
.run_cancellable(move |is_cancelled| Ok(prune_op(is_cancelled, true)))
.await
.map_err(to_file_error)?,
};
}
if recursive {
self.move_to_trash_with_namespace_owner(delete_path, recursive, immediate_purge, namespace_owner.clone())
.await?;
@@ -24563,6 +24661,20 @@ mod test {
assert!(!DurabilityMode::LegacyOff.syncs_recovery_protocol_state());
}
#[test]
#[serial_test::serial(file_fdatasync_group_commit)]
fn test_file_fdatasync_grouping_only_blocks_non_inline_fused_commit() {
let _grouping = os::set_file_fdatasync_group_commit_for_test(true);
assert!(
!commit::file_fdatasync_grouping_requires_async_shard_sync(false),
"inline metadata has no shard directory and must keep the fused path"
);
assert!(
commit::file_fdatasync_grouping_requires_async_shard_sync(true),
"non-inline shard sync must remain on the grouping-aware async path"
);
}
#[test]
fn test_system_critical_volume_classification() {
// System namespaces are pinned.
@@ -25395,14 +25507,223 @@ mod test {
.join(STORAGE_FORMAT_FILE_BACKUP)
.is_file()
);
use std::os::unix::fs::MetadataExt;
let v3_meta_bytes = fs::read(dst_object_dir.join(STORAGE_FORMAT_FILE))
.await
.expect("read v3 xl.meta");
let v3_meta_ino = std::fs::metadata(dst_object_dir.join(STORAGE_FORMAT_FILE))
.expect("stat v3 xl.meta")
.ino();
let synthetic_rollback = inline_metadata_rollback_dir(version_id, &FileMeta::new());
let synthetic_backup_parent = dst_object_dir.join(synthetic_rollback.to_string());
let synthetic_backup_path = synthetic_backup_parent.join(STORAGE_FORMAT_FILE_BACKUP);
fs::create_dir_all(&synthetic_backup_parent)
.await
.expect("create stale synthetic dir");
fs::write(&synthetic_backup_path, b"stale-bkp")
.await
.expect("write stale xl.meta.bkp");
let fi_v4 = test_file_info(object, version_id, None, Some(Bytes::from_static(b"v4-inline")));
let resp4 = disk
.rename_data(RUSTFS_META_TMP_BUCKET, "tmp-v4", fi_v4, bucket, object)
.await
.expect("fused inline overwrite v4 over inline v3");
assert_eq!(
(resp4.old_data_dir, resp4.rollback_data_dir, resp4.cleanup_data_dir),
(None, Some(synthetic_rollback), None)
);
assert_eq!(fs::read(&synthetic_backup_path).await.expect("read v3 synthetic backup"), v3_meta_bytes);
assert_eq!(
std::fs::metadata(&synthetic_backup_path).expect("stat v3 backup").ino(),
v3_meta_ino,
"linkat path must hardlink the previous xl.meta inode"
);
assert!(os::fsync_dir_recorder::was_fsynced(&synthetic_backup_parent));
assert!(os::fsync_dir_recorder::was_limited(&synthetic_backup_parent));
disk.delete(
bucket,
&format!("{object}/{synthetic_rollback}/{STORAGE_FORMAT_FILE_BACKUP}"),
DeleteOptions::default(),
)
.await
.expect("reclaim synthetic inline rollback backup");
assert!(!synthetic_backup_parent.exists(), "empty synthetic rollback dir must be pruned");
assert!(dst_object_dir.join(STORAGE_FORMAT_FILE).is_file());
let open_dst_meta = disk
.io_get_object_open_path(bucket, &format!("{object}/{STORAGE_FORMAT_FILE}"))
.expect("open dst meta path");
let v4_meta_bytes = fs::read(dst_object_dir.join(STORAGE_FORMAT_FILE))
.await
.expect("read v4 xl.meta");
let v4_meta_ino = std::fs::metadata(dst_object_dir.join(STORAGE_FORMAT_FILE))
.expect("stat v4 xl.meta")
.ino();
set_local_inline_rollback_hardlink_failure(&open_dst_meta);
fs::create_dir_all(&synthetic_backup_parent)
.await
.expect("create synthetic dir with child");
fs::write(synthetic_backup_parent.join("child-marker"), b"keep")
.await
.expect("write child marker");
let fi_v5 = test_file_info(object, version_id, None, Some(Bytes::from_static(b"v5-inline")));
let resp5 = disk
.rename_data(RUSTFS_META_TMP_BUCKET, "tmp-v5", fi_v5, bucket, object)
.await
.expect("fused inline overwrite v5 copy fallback");
assert_eq!(resp5.rollback_data_dir, Some(synthetic_rollback));
assert_eq!(
fs::read(&synthetic_backup_path).await.expect("read v4 copy-fallback backup"),
v4_meta_bytes
);
assert_ne!(
std::fs::metadata(&synthetic_backup_path).expect("stat v4 backup").ino(),
v4_meta_ino,
"copy-fallback path must allocate a new backup inode"
);
disk.delete(
bucket,
&format!("{object}/{synthetic_rollback}/{STORAGE_FORMAT_FILE_BACKUP}"),
DeleteOptions::default(),
)
.await
.expect("reclaim synthetic backup while child exists");
assert!(!synthetic_backup_path.exists());
assert!(
synthetic_backup_parent.join("child-marker").is_file(),
"non-empty synthetic rollback dir must preserve child entries"
);
fs::remove_file(synthetic_backup_parent.join("child-marker"))
.await
.expect("remove child marker");
fs::remove_dir(&synthetic_backup_parent)
.await
.expect("remove empty synthetic dir");
let legacy_key = "parent/xl.meta.bkp";
let legacy_fi = test_file_info(legacy_key, version_id, None, Some(Bytes::from_static(b"legacy-inline")));
let legacy_dir = disk.io_get_object_path(bucket, legacy_key).expect("legacy object dir");
fs::create_dir_all(&legacy_dir).await.expect("create legacy dir");
fs::write(legacy_dir.join(STORAGE_FORMAT_FILE), test_meta(legacy_fi.clone()))
.await
.expect("write legacy xl.meta");
disk.delete_version(bucket, legacy_key, legacy_fi, false, DeleteOptions::default())
.await
.expect("delete legacy key named xl.meta.bkp");
assert!(!disk.io_get_object_path(bucket, "parent").expect("parent path").exists());
let dst_meta_path = dst_object_dir.join(STORAGE_FORMAT_FILE);
let v5_meta_bytes = fs::read(&dst_meta_path).await.expect("read v5 xl.meta");
let dst_vol_dir = disk.io_get_bucket_path(bucket).expect("bucket path");
let lease = os::acquire_namespace_mutation_lease_with_owner(&dst_meta_path, None).await;
let guard =
commit::lock_rename_commit_directories(&dst_object_dir, &dst_object_dir, &dst_vol_dir, &disk.publication_root, lease)
.await
.expect("lock commit guard");
fs::create_dir_all(synthetic_backup_path.join("nested-child"))
.await
.expect("create conflicting directory at xl.meta.bkp");
assert!(
os::stage_inline_rollback_backup_with_commit_guard_std(
&dst_meta_path,
&synthetic_backup_parent,
&synthetic_backup_path,
&v5_meta_bytes,
true,
true,
&guard,
)
.is_err(),
"directory at xl.meta.bkp must fail staging without removing child"
);
assert!(synthetic_backup_path.join("nested-child").is_dir());
fs::remove_dir_all(&synthetic_backup_parent)
.await
.expect("remove conflicting synthetic dir");
os::fsync_dir_recorder::set_failure(&synthetic_backup_parent, std::io::ErrorKind::Other);
assert!(
os::stage_inline_rollback_backup_with_commit_guard_std(
&dst_meta_path,
&synthetic_backup_parent,
&synthetic_backup_path,
&v5_meta_bytes,
true,
true,
&guard,
)
.is_err()
);
assert!(!synthetic_backup_parent.exists(), "staging fsync failure must clean up created dir");
os::stage_inline_rollback_backup_with_commit_guard_std(
&dst_meta_path,
&synthetic_backup_parent,
&synthetic_backup_path,
&v5_meta_bytes,
true,
true,
&guard,
)
.expect("stage direct inline rollback backup");
assert!(synthetic_backup_path.is_file());
let displaced_object_dir = dst_object_dir.with_file_name("fused-object-displaced");
fs::rename(&dst_object_dir, &displaced_object_dir)
.await
.expect("displace dst_object_dir");
fs::create_dir_all(&synthetic_backup_parent)
.await
.expect("create replacement synthetic dir");
fs::write(&synthetic_backup_path, b"replacement-bkp")
.await
.expect("write replacement backup");
commit::discard_staged_inline_rollback_backup_std(synthetic_rollback, None, &guard);
assert_eq!(
fs::read(&synthetic_backup_path)
.await
.expect("replacement backup must survive"),
b"replacement-bkp",
"discard must operate on the pinned guard directory, not the replacement path"
);
assert!(
!displaced_object_dir.join(synthetic_rollback.to_string()).exists(),
"discard must remove the backup and synthetic dir from the pinned directory"
);
fs::remove_dir_all(&dst_object_dir)
.await
.expect("remove replacement object dir");
fs::rename(&displaced_object_dir, &dst_object_dir)
.await
.expect("restore original object dir");
os::stage_inline_rollback_backup_with_commit_guard_std(
&dst_meta_path,
&synthetic_backup_parent,
&synthetic_backup_path,
&v5_meta_bytes,
true,
true,
&guard,
)
.expect("stage direct inline rollback backup for fsync rollback");
let staged_new_meta = dst_object_dir.join("staged-new.xl.meta");
fs::write(&staged_new_meta, b"uncommitted-meta")
.await
.expect("write staged new meta");
fs::rename(&staged_new_meta, &dst_meta_path)
.await
.expect("replace dst_meta_path before fsync failure");
os::fsync_dir_recorder::set_failure(&dst_object_dir, std::io::ErrorKind::Other);
assert!(
commit::fsync_inline_commit_std(&dst_meta_path, &dst_vol_dir, false, Some(synthetic_rollback), None, None, &guard,)
.is_err()
);
assert!(
!synthetic_backup_parent.exists(),
"fsync rollback must remove consumed synthetic rollback dir"
);
assert_eq!(fs::read(&dst_meta_path).await.expect("restored v5"), v5_meta_bytes);
let replacement_dir = dst_object_dir.join(Uuid::new_v4().to_string());
fs::create_dir_all(&replacement_dir).await.expect("create replacement_dir");
fs::write(replacement_dir.join("part.1"), b"live").await.expect("write part");
@@ -37574,12 +37895,12 @@ mod test {
.expect("truncate behind cached descriptor");
assert!(read().await.is_err(), "native short data must not be accepted");
tokio::time::timeout(Duration::from_secs(5), async {
while backend.driver.stats().in_flight != 0 {
while backend.driver.stats().in_flight != 0 || memory.snapshot().used != 0 {
tokio::task::yield_now().await;
}
})
.await
.expect("failed native read retired");
.expect("failed native read retired and released its memory");
assert!(backend.driver.stats().submitted > submitted, "short read must reach the native driver");
assert_eq!(memory.snapshot().used, 0, "errors free native and assembled storage");
std::fs::write(dir.path().join("bucket/part.1"), &content).expect("restore fixture");
+45 -16
View File
@@ -231,14 +231,26 @@ fn rollback_metadata_std(destination_metadata: &Path, rollback_data_dir: Option<
let Some(dst_parent) = destination_metadata.parent() else {
return Err(std::io::Error::new(ErrorKind::InvalidInput, "missing object metadata parent"));
};
let backup_path = dst_parent.join(old_data_dir.to_string()).join(STORAGE_FORMAT_FILE_BACKUP);
std::fs::rename(backup_path, destination_metadata)?;
let rollback_parent = dst_parent.join(old_data_dir.to_string());
let backup_path = rollback_parent.join(STORAGE_FORMAT_FILE_BACKUP);
std::fs::rename(&backup_path, destination_metadata)?;
let _ = remove_file_if_exists(&backup_path);
let _ = std::fs::remove_dir(rollback_parent);
} else {
remove_file_if_exists(destination_metadata)?;
}
Ok(())
}
#[cfg(target_os = "linux")]
pub(super) fn discard_staged_inline_rollback_backup_std(
old_data_dir: Uuid,
has_old_data_dir: Option<Uuid>,
commit_guard: &os::RenameCommitGuard,
) {
os::discard_inline_rollback_backup_with_commit_guard_std(&old_data_dir.to_string(), has_old_data_dir.is_none(), commit_guard);
}
fn rollback_inline_metadata_std(
destination_metadata: &Path,
rollback_data_dir: Option<Uuid>,
@@ -412,6 +424,14 @@ enum FusedCommitAttempt {
ResumePanic(Box<dyn std::any::Any + Send + 'static>),
}
/// File fdatasync grouping only owns staged shard files. Inline commits have
/// no shard directory, so the switch must not evict them from the flat/fused
/// metadata path where grouping can never run.
#[cfg(any(target_os = "linux", test))]
pub(super) fn file_fdatasync_grouping_requires_async_shard_sync(no_inline: bool) -> bool {
no_inline && os::file_fdatasync_group_commit_enabled()
}
async fn prepare_rename_commit_and_destination_metadata(
create_source_parent: bool,
source_parent: &Path,
@@ -750,7 +770,6 @@ impl LocalDisk {
&& pool_meta_condition.is_none()
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !has_rename_data_test_failpoints(dst_path);
let src_file_path = match multipart_publication.as_ref() {
Some(publication) => publication.canonical_head(),
@@ -838,7 +857,7 @@ impl LocalDisk {
&& pool_meta_condition.is_none()
&& !fi_healing
&& !os::dst_dir_fsync_group_commit_enabled()
&& !os::file_fdatasync_group_commit_enabled()
&& !file_fdatasync_grouping_requires_async_shard_sync(no_inline)
{
#[cfg(test)]
if !no_inline {
@@ -961,9 +980,9 @@ impl LocalDisk {
}
let mut staged_rollback_path = None;
if (!no_inline || rollback_data_dir.is_none())
if rollback_data_dir.is_none()
&& let Some(ref old_metadata) = has_dst_buf
&& (rollback_data_dir.is_some() || cfg!(test))
&& cfg!(test)
{
let backup_path = match create_local_inline_rollback_backup(
&dst_file_path_owned,
@@ -1117,25 +1136,32 @@ impl LocalDisk {
if crash_inject::should_crash_at(CrashPoint::RenameAfterBackupBeforeMetaCommit, &dst_path_str) {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
} else if let (Some(old_data_dir), Some(staged_backup)) =
(rollback_data_dir, staged_rollback_path.as_ref().map(StagedRollbackBackup::path))
{
} else if let (Some(old_data_dir), Some(dst_buf)) = (rollback_data_dir, has_dst_buf.as_ref()) {
let backup_parent = dst_parent_owned.join(old_data_dir.to_string());
let backup_path = backup_parent.join(STORAGE_FORMAT_FILE_BACKUP);
if let Err(err) = os::rename_all_with_precondition_and_parent_sync_std(
staged_backup,
if let Err(err) = check_path_length(backup_path.to_string_lossy().as_ref()) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
return Ok(FusedCommitAttempt::CommitErr(err));
}
#[cfg(test)]
run_owned_file_before_open(&backup_path);
if let Err(err) = os::stage_inline_rollback_backup_with_commit_guard_std(
&dst_file_path_owned,
&backup_parent,
&backup_path,
&skip_parent,
&publication_root_owned,
None,
dst_buf.as_ref(),
sync_recovery_protocol,
!should_fail_local_inline_rollback_hardlink(&dst_file_path_owned),
&guard,
) {
drop(prepared_metadata_source);
cleanup_flat_tmp();
let _ = remove_file_if_exists(staged_backup);
return Ok(FusedCommitAttempt::CommitErr(to_file_error(err).into()));
}
staged_rollback_path = None;
if crash_inject::should_crash_at(CrashPoint::RenameAfterBackupBeforeMetaCommit, &dst_path_str) {
return Ok(FusedCommitAttempt::CommitErr(DiskError::Unexpected));
}
}
let rename_res = if should_fail_commit_rename(&dst_path_str) {
@@ -1159,6 +1185,9 @@ impl LocalDisk {
if no_inline {
return Ok(FusedCommitAttempt::RestoreDataSource(guard, err));
}
if let Some(old_data_dir) = rollback_data_dir {
discard_staged_inline_rollback_backup_std(old_data_dir, has_old_data_dir, &guard);
}
return Ok(FusedCommitAttempt::CommitErr(err));
}
File diff suppressed because it is too large Load Diff
@@ -4009,6 +4009,7 @@ impl SetDisks {
file_info.validate_for_erasure_write()?;
}
}
let file_infos = Arc::new(file_infos);
let disk_count = disks.len();
let fanout_disks = disks.to_vec();
@@ -4031,19 +4032,16 @@ impl SetDisks {
let successful_rename_completion_rank =
rustfs_io_metrics::put_stage_metrics_enabled().then(|| Arc::new(AtomicUsize::new(0)));
let mut tasks = JoinSet::new();
for (i, ((disk, file_info), scanner_publication_lease_token)) in fanout_disks
.into_iter()
.zip(file_infos.iter())
.zip(fanout_fence_tokens.into_iter())
.enumerate()
for (i, (disk, scanner_publication_lease_token)) in
fanout_disks.into_iter().zip(fanout_fence_tokens.into_iter()).enumerate()
{
let src_bucket = fanout_src_bucket.clone();
let src_object = fanout_src_object.clone();
let dst_bucket = fanout_dst_bucket.clone();
let dst_object = fanout_dst_object.clone();
let file_info = file_info.clone();
let successful_rename_completion_rank = successful_rename_completion_rank.clone();
let namespace_commit_guard = namespace_commit_guard.clone();
let file_infos = file_infos.clone();
tasks.spawn(async move {
let _namespace_commit_guard = namespace_commit_guard.clone();
let mut dispatch_state = RenameDispatchState::NotDispatched;
@@ -4060,12 +4058,16 @@ impl SetDisks {
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT_REMOTE
};
let file_info = &file_infos[i];
let is_delete_marker = file_info.is_canonical_delete_marker();
// Clone FileInfo and set erasure.index for this disk
let mut file_info = file_info.clone();
if file_info.erasure.index == 0 {
file_info.erasure.index = i + 1;
}
let mut indexed_file_info;
let file_info = if file_info.erasure.index == 0 {
indexed_file_info = file_info.clone();
indexed_file_info.erasure.index = i + 1;
&indexed_file_info
} else {
file_info
};
if file_info.erasure.index == 0 || (!is_delete_marker && !file_info.has_valid_erasure_geometry()) {
return Err(DiskError::FileCorrupt);
}
@@ -4083,7 +4085,7 @@ impl SetDisks {
.rename_data_borrowed_with_fence_observed(
&src_bucket,
&src_object,
&file_info,
file_info,
&dst_bucket,
&dst_object,
disk::RenameDataGuards {
@@ -4191,7 +4193,7 @@ impl SetDisks {
let snapshot_commit = Self::rename_data_commit_from_observations(
RenameDataObservations {
disks: &coordinator_disks,
file_infos: &file_infos,
file_infos: file_infos.as_slice(),
disk_versions: &disk_versions,
errs: &errs,
cleanup_data_dirs: &cleanup_data_dirs,
@@ -4236,6 +4238,7 @@ impl SetDisks {
if !sent_commit {
let ret_err =
reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum).unwrap_or(DiskError::Unexpected);
let file_infos = Arc::try_unwrap(file_infos).unwrap_or_else(|file_infos| file_infos.as_ref().clone());
rollback_failed_rename(
&coordinator_disks,
file_infos,
+68
View File
@@ -866,6 +866,74 @@ where
}
let _mutation_guard = self.cache.service_account_mutation_lock().lock().await;
self.save_new_service_account_locked(cred, updated_at).await
}
/// Order a cross-user issue with changes to either identity. Parent locks
/// are acquired by name, then the service-account lock, then the caller's
/// STS lock, matching user updates. Hold them through durable publication.
pub(crate) async fn add_delegated_service_account(&self, cred: &Credentials, caller: &Credentials) -> Result<OffsetDateTime> {
use subtle::ConstantTimeEq;
let target = cred.parent_user.as_str();
let principal = if caller.parent_user.is_empty() {
caller.access_key.as_str()
} else {
caller.parent_user.as_str()
};
if target == principal || target == caller.access_key || crate::is_root_access_key(target) {
return Err(Error::IAMActionNotAllowed);
}
let mut parents = [principal, target];
parents.sort_unstable();
let _first = self.cache.parent_mutation_lock(parents[0]).lock_owned().await;
let _second = self.cache.parent_mutation_lock(parents[1]).lock_owned().await;
let _service = self.cache.service_account_mutation_lock().lock().await;
let _session = if !caller.session_token.is_empty() && !caller.is_service_account() {
Some(self.cache.sts_account_mutation_lock(&caller.access_key).lock_owned().await)
} else {
None
};
let active_local_user = |user: &UserIdentity| {
user.credentials.is_valid() && user.credentials.parent_user.is_empty() && user.credentials.session_token.is_empty()
};
let snapshot = self.cache.snapshot();
if !snapshot.users.get(target).is_some_and(active_local_user) {
return Err(Error::IAMActionNotAllowed);
}
let root = crate::root_credentials::credentials().ok_or(Error::IAMActionNotAllowed)?;
let current = if caller.access_key == root.access_key {
&root
} else {
&snapshot
.users
.get(&caller.access_key)
.or_else(|| snapshot.sts_accounts.get(&caller.access_key))
.ok_or(Error::IAMActionNotAllowed)?
.credentials
};
if !current.is_valid()
|| current.parent_user != caller.parent_user
|| !bool::from(current.secret_key.as_bytes().ct_eq(caller.secret_key.as_bytes()))
|| !bool::from(current.session_token.as_bytes().ct_eq(caller.session_token.as_bytes()))
{
return Err(Error::IAMActionNotAllowed);
}
if !caller.parent_user.is_empty()
&& principal != root.access_key
&& !crate::sys::is_rustfs_oidc_claims(caller.claims_or_empty())
&& !snapshot.users.get(principal).is_some_and(active_local_user)
{
return Err(Error::IAMActionNotAllowed);
}
drop(snapshot);
self.save_new_service_account_locked(cred.clone(), OffsetDateTime::now_utc())
.await
}
/// The caller holds the service-account mutation lock through cache publication.
async fn save_new_service_account_locked(&self, cred: Credentials, updated_at: OffsetDateTime) -> Result<OffsetDateTime> {
let cache = self.cache.snapshot();
if cache.users.contains_key(&cred.access_key) || cache.sts_accounts.contains_key(&cred.access_key) {
return Err(Error::AccessKeyAlreadyExists);
+321 -8
View File
@@ -960,6 +960,19 @@ impl<T: Store> IamSys<T> {
opts: NewServiceAccountOpts,
updated_at: OffsetDateTime,
) -> Result<(Credentials, OffsetDateTime)> {
let cred = Self::service_account_credentials(parent_user, groups, opts)?;
let create_at = self.store.add_service_account_at(cred.clone(), updated_at).await?;
self.notify_for_service_account(&cred.access_key).await;
Ok((cred, create_at))
}
fn service_account_credentials(
parent_user: &str,
groups: Option<Vec<String>>,
opts: NewServiceAccountOpts,
) -> Result<Credentials> {
if !opts.access_key.is_empty()
&& (opts.access_key.trim() == "."
|| opts.access_key.trim() == ".."
@@ -1046,11 +1059,7 @@ impl<T: Store> IamSys<T> {
cred.name = opts.name;
cred.description = opts.description;
let create_at = self.store.add_service_account_at(cred.clone(), updated_at).await?;
self.notify_for_service_account(&cred.access_key).await;
Ok((cred, create_at))
Ok(cred)
}
pub async fn update_service_account(&self, name: &str, opts: UpdateServiceAccountOpts) -> Result<OffsetDateTime> {
@@ -1689,6 +1698,80 @@ impl<T: Store> IamSys<T> {
Ok(Some(verified_claims))
}
/// Issue a credential for a different local user. Both actions must allow the
/// authenticated caller and the same resolved target. Once admitted, the
/// write continues if the caller stops waiting.
///
/// Returns an error if the caller lacks either grant, the target is missing
/// or ineligible, the caller has been revoked, or the key cannot be saved.
pub async fn new_service_account_for_user(
&self,
target: &str,
opts: NewServiceAccountOpts,
caller: &Credentials,
conditions: &HashMap<String, Vec<String>>,
authenticated_owner: bool,
) -> Result<(Credentials, OffsetDateTime, HashMap<String, Value>)> {
use rustfs_policy::policy::{
SVC_TARGET_USER,
action::{Action, AdminAction},
};
if target.is_empty() || crate::is_root_access_key(target) || target == caller.access_key || target == caller.parent_user {
return Err(IamError::IAMActionNotAllowed);
}
// Bind the policy decision to the identity that will be written.
if !conditions
.get(SVC_TARGET_USER)
.is_some_and(|values| values.len() == 1 && values[0] == target)
{
return Err(IamError::IAMActionNotAllowed);
}
let mut args = Args {
account: &caller.access_key,
groups: &caller.groups,
action: Action::AdminAction(AdminAction::CreateServiceAccountAdminAction),
bucket: "",
conditions,
is_owner: authenticated_owner,
object: "",
claims: caller.claims_or_empty(),
deny_only: false,
};
// Both built-in decisions must use the same identity and policy revision.
let prepared = self.prepare_auth(&args).await;
if !self.eval_prepared(&prepared, &args).await {
return Err(IamError::IAMActionNotAllowed);
}
args.action = Action::AdminAction(AdminAction::CreateServiceAccountForUserAdminAction);
if !self.eval_prepared(&prepared, &args).await {
return Err(IamError::IAMActionNotAllowed);
}
if self.check_key(target).await?.0.is_none() {
return Err(IamError::NoSuchUser(target.to_owned()));
}
// A delegated credential belongs entirely to its local target. Only
// its requested session policy and expiration constrain that identity.
let opts = NewServiceAccountOpts {
claims: None,
allow_site_replicator_account: false,
..opts
};
let cred = Self::service_account_credentials(target, None, opts)?;
let iam = self.clone();
let caller = caller.clone();
let runtime = tokio::runtime::Handle::try_current().map_err(Error::other)?;
runtime
.spawn(async move {
let updated_at = iam.store.add_delegated_service_account(&cred, &caller).await?;
iam.notify_for_service_account(&cred.access_key).await;
Ok((cred, updated_at, HashMap::new()))
})
.await
.map_err(Error::other)?
}
/// Create within the authenticated caller's scope and current parent revision.
/// An admitted write owns its parent lock until publication even if the caller
/// stops waiting; errors never publish a credential from a changed parent.
@@ -1719,8 +1802,7 @@ impl<T: Store> IamSys<T> {
mut opts: NewServiceAccountOpts,
caller: &Credentials,
) -> Result<(Credentials, OffsetDateTime, HashMap<String, Value>)> {
if parent_user != caller.access_key && parent_user != caller.parent_user && !crate::is_root_access_key(&caller.access_key)
{
if parent_user != caller.access_key && parent_user != caller.parent_user {
return Err(IamError::IAMActionNotAllowed);
}
@@ -3347,6 +3429,7 @@ mod tests {
/// When true, parent user has no groups and no mapped policies (empty `policy_db_get`).
empty_policies: bool,
bucket_tags: Arc<Mutex<Result<HashMap<String, String>>>>,
policy_load_barrier: Option<Arc<tokio::sync::Barrier>>,
saved_sts_users: Arc<Mutex<HashMap<String, UserIdentity>>>,
saved_service_account_count: Arc<Mutex<usize>>,
fail_delete: Arc<std::sync::atomic::AtomicBool>,
@@ -3369,6 +3452,7 @@ mod tests {
fn new(empty_policies: bool) -> Self {
Self {
empty_policies,
policy_load_barrier: None,
bucket_tags: Arc::new(Mutex::new(Err(Error::other("bucket metadata unavailable")))),
saved_sts_users: Arc::new(Mutex::new(HashMap::new())),
saved_service_account_count: Arc::new(Mutex::new(0)),
@@ -3561,7 +3645,14 @@ mod tests {
Err(Error::InvalidArgument)
}
async fn load_policy_doc(&self, _name: &str, _m: &mut HashMap<String, rustfs_policy::policy::PolicyDoc>) -> Result<()> {
async fn load_policy_doc(&self, name: &str, docs: &mut HashMap<String, PolicyDoc>) -> Result<()> {
if let Some(barrier) = &self.policy_load_barrier {
// merge_policies has already captured the cached policy documents.
barrier.wait().await;
barrier.wait().await;
docs.insert(name.to_owned(), PolicyDoc::new(Policy::default()));
return Ok(());
}
Err(Error::InvalidArgument)
}
@@ -4391,6 +4482,67 @@ mod tests {
}
}
#[tokio::test]
async fn delegated_service_account_holds_both_parents_and_session_through_publication() {
use std::sync::atomic::Ordering;
ensure_test_global_credentials();
let store = StsTestMockStore::new(false);
let iam = IamSys::new(IamCache::new(store.clone()).await.expect("initialize IAM"));
let caller_parent = "operator";
let target = "backup";
for user in [caller_parent, target] {
iam.create_user(
user,
&AddOrUpdateUserReq {
secret_key: "originalSecret123".into(),
status: rustfs_madmin::AccountStatus::Enabled,
policy: None,
},
)
.await
.expect("create local user");
}
let session = rotation_test_session(caller_parent, "operator-console-session");
iam.set_temp_user(&session.access_key, &session, None)
.await
.expect("publish STS session");
store.block_derived_save.store(true, Ordering::SeqCst);
let pending = IamSys::<StsTestMockStore>::service_account_credentials(target, None, NewServiceAccountOpts::default())
.expect("prepare credential");
let creation = {
let iam = iam.clone();
let session = session.clone();
tokio::spawn(async move { iam.store.add_delegated_service_account(&pending, &session).await })
};
tokio::time::timeout(std::time::Duration::from_secs(5), store.derived_save_started.notified())
.await
.expect("credential write started");
for principal in [caller_parent, target] {
assert!(iam.store.cache.parent_mutation_lock(principal).try_lock().is_err());
}
assert!(
iam.store
.cache
.sts_account_mutation_lock(&session.access_key)
.try_lock()
.is_err()
);
store.block_derived_save.store(false, Ordering::SeqCst);
store.release_derived_save.notify_one();
creation.await.expect("join writer").expect("publish credential");
iam.set_user_status(target, rustfs_madmin::AccountStatus::Disabled)
.await
.expect("disable target");
let pending = IamSys::<StsTestMockStore>::service_account_credentials(target, None, NewServiceAccountOpts::default())
.expect("prepare credential for disabled user");
assert!(matches!(
iam.store.add_delegated_service_account(&pending, &session).await,
Err(Error::IAMActionNotAllowed)
));
assert_eq!(store.saved_service_account_count(), 1);
}
#[tokio::test]
async fn revoked_sts_cannot_mint_service_account_between_rotation_and_sweep() {
ensure_test_global_credentials();
@@ -5332,6 +5484,167 @@ mod tests {
);
}
#[tokio::test]
async fn delegated_service_account_requires_both_actions_in_one_policy_revision() {
use rustfs_policy::policy::SVC_TARGET_USER;
ensure_test_global_credentials();
let barrier = Arc::new(tokio::sync::Barrier::new(2));
let mut store = StsTestMockStore::new(true);
store.policy_load_barrier = Some(barrier.clone());
let cache = temp_env::async_with_vars([("RUSTFS_SKIP_BACKGROUND_TASK", Some("1"))], IamCache::new(store.clone()))
.await
.expect("initialize IAM");
let iam = IamSys::new(cache);
let operator = "backup-operator";
let target = "backup-user";
for name in [operator, target] {
iam.create_user(
name,
&AddOrUpdateUserReq {
secret_key: "testSecret123".into(),
status: rustfs_madmin::AccountStatus::Enabled,
policy: None,
},
)
.await
.expect("create local user");
}
let caller = iam.get_user(operator).await.expect("operator exists").credentials;
let set_grant = |action: &str| {
let policy = serde_json::json!({"Version": "2012-10-17", "Statement": [{"Effect": "Allow", "Action": action}]});
let policy = Policy::parse_config(policy.to_string().as_bytes()).expect("parse administrator policy");
iam.store
.cache
.add_or_update_policy_doc("operator-grant", &PolicyDoc::new(policy), OffsetDateTime::now_utc());
};
set_grant("admin:CreateServiceAccount");
iam.store.cache.add_or_update_user_policy(
operator,
&MappedPolicy::new("operator-grant,uncached-policy"),
OffsetDateTime::now_utc(),
);
let conditions = HashMap::from([(SVC_TARGET_USER.to_owned(), vec![target.to_owned()])]);
let create = iam.new_service_account_for_user(target, NewServiceAccountOpts::default(), &caller, &conditions, false);
let replace = async {
barrier.wait().await;
set_grant("admin:CreateServiceAccountForUser");
barrier.wait().await;
};
let (result, ()) = tokio::time::timeout(std::time::Duration::from_secs(5), async { tokio::join!(create, replace) })
.await
.expect("finish lookup and concurrent policy replacement");
assert!(matches!(result, Err(Error::IAMActionNotAllowed)));
assert_eq!(store.saved_service_account_count(), 0);
}
#[tokio::test]
async fn delegated_service_account_distinguishes_missing_target_from_storage_failure() {
use rustfs_policy::policy::SVC_TARGET_USER;
ensure_test_global_credentials();
let store = StsTestMockStore::new(true);
let iam = IamSys::new(IamCache::new(store.clone()).await.expect("initialize IAM"));
let caller = crate::root_credentials::credentials().expect("root credentials");
for target in ["missing-target", "load-failure-user"] {
let conditions = HashMap::from([(SVC_TARGET_USER.to_owned(), vec![target.to_owned()])]);
let error = iam
.new_service_account_for_user(target, NewServiceAccountOpts::default(), &caller, &conditions, true)
.await
.expect_err("cannot publish without a target");
match target {
"missing-target" => assert!(matches!(error, Error::NoSuchUser(name) if name == target)),
_ => assert!(matches!(error, Error::Io(_))),
}
}
assert_eq!(store.saved_service_account_count(), 0);
}
#[tokio::test]
async fn delegated_service_account_uses_local_target_permissions_and_requires_live_oidc_session() {
use rustfs_policy::policy::{
SVC_TARGET_USER,
action::{Action, AdminAction, S3Action},
};
ensure_test_global_credentials();
// This fixture seeds a policy mapping in memory. Its mock reload only
// persists identities and would replace the mapping with its seed data.
let cache =
temp_env::async_with_vars([("RUSTFS_SKIP_BACKGROUND_TASK", Some("1"))], IamCache::new(StsTestMockStore::new(true)))
.await
.expect("initialize IAM");
let iam = IamSys::new(cache);
let target = "backup-user";
iam.create_user(
target,
&AddOrUpdateUserReq {
secret_key: "backupUserSecret123".into(),
status: rustfs_madmin::AccountStatus::Enabled,
policy: None,
},
)
.await
.expect("create backup user");
iam.store
.cache
.add_or_update_user_policy(target, &MappedPolicy::new("readonly"), OffsetDateTime::now_utc());
let mut caller = rotation_test_session("oidc-admin-parent", "oidc-console-session");
caller.claims = Some(HashMap::from([
("iss".into(), Value::from("rustfs-oidc")),
("oidc_provider".into(), Value::from("default")),
("sub".into(), Value::from("operator")),
("parent".into(), Value::from(caller.parent_user.clone())),
(OIDC_VIRTUAL_PARENT_CLAIM.into(), Value::from(caller.parent_user.clone())),
(POLICYNAME.into(), Value::from("consoleAdmin")),
]));
iam.set_temp_user(&caller.access_key, &caller, None)
.await
.expect("publish OIDC session");
let conditions = HashMap::from([(SVC_TARGET_USER.to_owned(), vec![target.to_owned()])]);
let (created, _, replicated_claims) = iam
.new_service_account_for_user(target, NewServiceAccountOpts::default(), &caller, &conditions, false)
.await
.expect("delegate to local user");
assert_eq!(created.parent_user, target);
assert!(replicated_claims.is_empty());
let key = iam
.get_user(&created.access_key)
.await
.expect("published service account")
.credentials;
assert!(!is_rustfs_oidc_claims(key.claims_or_empty()));
let mut args = Args {
account: &key.access_key,
groups: &key.groups,
action: Action::S3Action(S3Action::GetObjectAction),
bucket: "backups",
object: "archive",
conditions: &HashMap::new(),
claims: key.claims_or_empty(),
is_owner: false,
deny_only: false,
};
assert!(iam.is_allowed(&args).await);
args.action = Action::S3Action(S3Action::PutObjectAction);
assert!(!iam.is_allowed(&args).await);
args.action = Action::AdminAction(AdminAction::CreateServiceAccountForUserAdminAction);
assert!(!iam.is_allowed(&args).await);
iam.store
.delete_user(&caller.access_key, UserType::Sts)
.await
.expect("revoke session");
let pending = IamSys::<StsTestMockStore>::service_account_credentials(target, None, NewServiceAccountOpts::default())
.expect("prepare credential");
assert!(matches!(
iam.store.add_delegated_service_account(&pending, &caller).await,
Err(Error::IAMActionNotAllowed)
));
assert!(iam.store.get_user(&pending.access_key).await.is_none());
}
#[tokio::test]
async fn oidc_service_account_uses_verified_policy_and_persisted_boundary() {
ensure_test_global_credentials();
@@ -454,7 +454,6 @@ const INTERNODE_OPERATION_LARGE_PAYLOADS_TOTAL: &str = "rustfs_system_network_in
const INTERNODE_MSGPACK_JSON_DECODE_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_decode_total";
const INTERNODE_MSGPACK_JSON_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_fallback_total";
const INTERNODE_MSGPACK_JSON_DECODE_ERROR_TOTAL: &str = "rustfs_system_network_internode_msgpack_json_decode_error_total";
const INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL: &str = "rustfs_system_network_internode_msgpack_send_decisions_total";
const INTERNODE_SIGNATURE_V1_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_signature_v1_fallback_total";
const INTERNODE_BODY_DIGEST_FALLBACK_TOTAL: &str = "rustfs_system_network_internode_body_digest_fallback_total";
@@ -1605,25 +1604,6 @@ impl InternodeMetrics {
.increment(1);
}
/// Record one eligible request's fixed-cardinality JSON/msgpack send decision.
pub fn record_msgpack_send_decision(&self, decision: &'static str, reason: &'static str) {
let decision = match decision {
"msgpack_only" | "dual_write" => decision,
_ => "dual_write",
};
let reason = match reason {
"manual_fleet" | "peer_capable" | "peer_unknown" | "peer_revoked" | "not_requested" => reason,
_ => "peer_unknown",
};
counter!(
INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL,
SERVER_LABEL => current_server_label(),
"decision" => decision,
REASON_LABEL => reason
)
.increment(1);
}
#[doc(hidden)]
pub fn msgpack_json_decode_error_total_for_test(&self) -> u64 {
self.msgpack_json_decode_error_total.load(Ordering::Relaxed)
@@ -2720,52 +2700,6 @@ mod tests {
assert!(observed.contains(&(INTERNODE_MSGPACK_DIRECTION_RESPONSE.to_string(), "RawFileInfo".to_string(), 1)));
}
#[test]
fn msgpack_send_decision_counter_keeps_fixed_decision_and_reason_labels() {
let recorder = DebuggingRecorder::new();
let snapshotter = recorder.snapshotter();
let metrics = InternodeMetrics::default();
metrics::with_local_recorder(&recorder, || {
for (decision, reason) in [
("msgpack_only", "manual_fleet"),
("msgpack_only", "peer_capable"),
("dual_write", "peer_unknown"),
("dual_write", "peer_revoked"),
("dual_write", "not_requested"),
] {
metrics.record_msgpack_send_decision(decision, reason);
}
metrics.record_msgpack_send_decision("untrusted", "untrusted");
});
let observed: HashMap<(String, String), u64> = snapshotter
.snapshot()
.into_vec()
.into_iter()
.filter(|(composite, _, _, _)| composite.key().name() == INTERNODE_MSGPACK_SEND_DECISIONS_TOTAL)
.map(|(composite, _, _, value)| {
let labels: HashMap<String, String> = composite
.key()
.labels()
.map(|label| (label.key().to_string(), label.value().to_string()))
.collect();
let count = match value {
DebugValue::Counter(count) => count,
other => panic!("msgpack send decision must be a counter, got {other:?}"),
};
(
(
labels.get("decision").cloned().unwrap_or_default(),
labels.get(REASON_LABEL).cloned().unwrap_or_default(),
),
count,
)
})
.collect();
assert_eq!(observed.len(), 5);
assert_eq!(observed.get(&("dual_write".to_string(), "peer_unknown".to_string())), Some(&2));
}
#[test]
fn msgpack_json_decode_counter_tracks_codec_direction_and_message() {
let metrics = InternodeMetrics::default();
+5 -1
View File
@@ -435,6 +435,7 @@ pub const PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL: &str = "serial";
pub const PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL: &str = "parallel";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER: &str = "leader";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER: &str = "follower";
pub const PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT: &str = "follower_timeout";
pub const PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_ENQUEUE_WAITERS: &str = "enqueue_waiters";
pub const PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_ENQUEUE_FILES: &str = "enqueue_files";
pub const PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_BATCH_WAITERS: &str = "batch_waiters";
@@ -4422,6 +4423,7 @@ mod tests {
set_put_stage_metrics_enabled(false);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_SERIAL, 2);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER, 1.0);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT, 3.0);
record_put_rename_fdatasync_group_outstanding(PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_ENQUEUE_WAITERS, 2);
record_put_rename_disk_wait_completion(PUT_RENAME_DISK_WAIT_COMPLETION_POSITION_QUORUM_FIRST, 3.0);
record_put_rename_quorum_wait_fanout(4, 3, 3, 1, 0);
@@ -4429,6 +4431,7 @@ mod tests {
set_put_stage_metrics_enabled(true);
record_put_rename_fdatasync_batch(PUT_RENAME_FDATASYNC_BATCH_MODE_PARALLEL, 9);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER, 1.0);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT, 3.0);
record_put_rename_fdatasync_group_wait(PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER, 2.0);
record_put_rename_fdatasync_group_outstanding(PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_ENQUEUE_WAITERS, 2);
record_put_rename_fdatasync_group_outstanding(PUT_RENAME_FDATASYNC_GROUP_OUTSTANDING_STATE_ENQUEUE_FILES, 4);
@@ -4464,7 +4467,7 @@ mod tests {
assert_eq!(quorum_samples, vec![0.0, 1.0, 3.0, 3.0, 4.0]);
assert_eq!(
histogram_samples(&rows, "rustfs_s3_put_object_rename_fdatasync_group_wait_ms"),
vec![1.0, 2.0]
vec![1.0, 2.0, 3.0]
);
assert_eq!(
histogram_samples(&rows, "rustfs_s3_put_object_rename_fdatasync_group_outstanding"),
@@ -4494,6 +4497,7 @@ mod tests {
HashSet::from([
PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_LEADER.to_string(),
PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER.to_string(),
PUT_RENAME_FDATASYNC_GROUP_WAIT_ROLE_FOLLOWER_TIMEOUT.to_string(),
])
);
let group_outstanding_states = rows
+1
View File
@@ -30,6 +30,7 @@ pub mod variables;
pub use action::ActionSet;
pub use doc::PolicyDoc;
pub use effect::Effect;
pub use function::key_name::SVC_TARGET_USER;
pub use function::{Functions, is_server_derived_condition_key};
pub use id::ID;
pub use policy::*;
+3
View File
@@ -495,6 +495,8 @@ pub enum AdminAction {
SiteReplicationOperationAction,
#[strum(serialize = "admin:CreateServiceAccount")]
CreateServiceAccountAdminAction,
#[strum(serialize = "admin:CreateServiceAccountForUser")]
CreateServiceAccountForUserAdminAction,
#[strum(serialize = "admin:UpdateServiceAccount")]
UpdateServiceAccountAdminAction,
#[strum(serialize = "admin:RemoveServiceAccount")]
@@ -705,6 +707,7 @@ impl AdminAction {
| AdminAction::SiteReplicationInfoAction
| AdminAction::SiteReplicationOperationAction
| AdminAction::CreateServiceAccountAdminAction
| AdminAction::CreateServiceAccountForUserAdminAction
| AdminAction::UpdateServiceAccountAdminAction
| AdminAction::RemoveServiceAccountAdminAction
| AdminAction::ListServiceAccountsAdminAction
+10 -1
View File
@@ -48,6 +48,9 @@ impl TryFrom<&str> for KeyName {
}
}
/// Canonical condition-map key for the resolved service-account parent.
pub const SVC_TARGET_USER: &str = "TargetUser";
impl KeyName {
pub const COMMON_KEYS: &'static [KeyName] = &[
// s3
@@ -136,7 +139,11 @@ impl KeyName {
/// Names (prefix stripped) of the well-known condition keys that request input
/// must never populate. See [`KeyName::is_server_derived`].
pub fn server_derived_key_names() -> impl Iterator<Item = &'static str> {
Self::COMMON_KEYS.iter().filter(|k| k.is_server_derived()).map(|k| k.name())
Self::COMMON_KEYS
.iter()
.filter(|k| k.is_server_derived())
.map(|k| k.name())
.chain([SVC_TARGET_USER])
}
/// Returns the policy variable form of the key, e.g. `${jwt:sub}`.
@@ -328,6 +335,8 @@ pub enum JwtKeyName {
pub enum SvcKeyName {
#[strum(serialize = "svc:DurationSeconds")]
SVCDurationSeconds,
#[strum(serialize = "svc:TargetUser")]
TargetUser,
}
#[derive(Clone, EnumString, Debug, IntoStaticStr, Eq, PartialEq, Serialize, Deserialize)]
+52
View File
@@ -644,6 +644,58 @@ mod test {
use crate::error::Result;
use crate::policy::action::{AdminAction, KmsAction, S3Action};
#[tokio::test]
async fn delegated_service_account_obeys_admin_wildcards_and_target_conditions() -> Result<()> {
let conditions = HashMap::from([("TargetUser".into(), vec!["backup".into()])]);
let args = Args {
account: "operator",
groups: &None,
action: Action::AdminAction(AdminAction::CreateServiceAccountForUserAdminAction),
bucket: "",
object: "",
conditions: &conditions,
claims: &HashMap::new(),
is_owner: false,
deny_only: false,
};
for (statement, allowed) in [
(serde_json::json!({"Effect":"Allow","Action":"admin:*"}), true),
(serde_json::json!({"Effect":"Allow","Action":"admin:CreateServiceAccount"}), false),
(serde_json::json!({"Effect":"Allow","NotAction":"admin:DeleteUser","Resource":"*"}), true),
(
serde_json::json!({"Effect":"Allow","NotAction":"admin:CreateServiceAccountForUser","Resource":"*"}),
false,
),
(
serde_json::json!({"Effect":"Allow","Action":"admin:CreateServiceAccountForUser","Condition":{"StringEquals":{"svc:TargetUser":"backup"}}}),
true,
),
(
serde_json::json!({"Effect":"Allow","Action":"admin:CreateServiceAccountForUser","Condition":{"StringEquals":{"svc:TargetUser":"archive"}}}),
false,
),
] {
let policy = Policy::parse_config(
&serde_json::to_vec(&serde_json::json!({"Version":DEFAULT_VERSION,"Statement":[statement]}))
.expect("encode policy"),
)?;
assert_eq!(policy.is_allowed(&args).await, allowed, "{statement}");
}
let admin = default::DEFAULT_POLICIES
.iter()
.find(|(name, _)| *name == "consoleAdmin")
.expect("built-in admin policy");
assert!(admin.1.is_allowed(&args).await);
for denied_action in ["admin:*", "admin:CreateServiceAccountForUser"] {
let deny = Policy::parse_config(&serde_json::to_vec(&serde_json::json!({
"Version": DEFAULT_VERSION,
"Statement": [{"Effect": "Deny", "Action": denied_action}]
}))?)?;
assert!(!Policy::merge_policies(vec![admin.1.clone(), deny]).is_allowed(&args).await);
}
Ok(())
}
#[tokio::test]
async fn test_parse_policy() -> Result<()> {
let data = r#"
+7 -7
View File
@@ -73,7 +73,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "batch_read_version_req",
bin_field: "batch_read_version_req_bin",
},
json_encoder: "let batch_read_version_req = compat_json(batch_rpc_req.as_ref(), &policy)?;",
json_encoder: "let batch_read_version_req = compat_json(&req)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -118,7 +118,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "read_multiple_req",
bin_field: "read_multiple_req_bin",
},
json_encoder: "let read_multiple_req = compat_json(&req, &policy)?;",
json_encoder: "let read_multiple_req = compat_json(&req)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -127,7 +127,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "opts",
bin_field: "opts_bin",
},
json_encoder: "let opts_str = compat_json(opts, &policy)?;",
json_encoder: "let encoded_opts = compat_json(opts).and_then(|opts_str| encode_msgpack(opts).map(|opts_bin| (opts_str, opts_bin)));",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -136,7 +136,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
json_encoder: "let file_info = compat_json(&fi)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -145,7 +145,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
json_encoder: "let file_info = compat_json(&fi)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -154,7 +154,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "opts",
bin_field: "opts_bin",
},
json_encoder: "let opts_str = compat_json(opts, &policy)?;",
json_encoder: "let opts_str = compat_json(&opts)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
RequestCompatSendSite {
@@ -163,7 +163,7 @@ pub const REQUEST_COMPAT_SEND_SITES: &[RequestCompatSendSite] = &[
json_field: "file_info",
bin_field: "file_info_bin",
},
json_encoder: "let file_info = compat_json(&fi, &policy)?;",
json_encoder: "let file_info = compat_json(&fi)?;",
policy: RequestJsonPolicy::MsgpackOnlyEligible,
},
];
-54
View File
@@ -141,7 +141,6 @@ impl Drop for DialLock {
}
static INTERNODE_RPC_MSGPACK_ONLY_CACHE: AtomicU8 = AtomicU8::new(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET);
static INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE: AtomicU8 = AtomicU8::new(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET);
fn enforce_tls_generation_cache_bound(generation_cache: &mut HashMap<String, u64>, generation: u64, addr: &str) {
if generation_cache.len() < TLS_GENERATION_CACHE_MAX_SIZE || generation_cache.contains_key(addr) {
@@ -3423,35 +3422,9 @@ pub fn internode_rpc_msgpack_only() -> bool {
enabled
}
/// Whether msgpack-only was requested and the default-off per-peer capability gate is enabled.
pub fn internode_rpc_msgpack_peer_capability_enabled() -> bool {
match INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.load(Ordering::Acquire) {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_FALSE => return false,
INTERNODE_RPC_MSGPACK_ONLY_CACHE_TRUE => return true,
_ => {}
}
let enabled = rustfs_utils::get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_MSGPACK_ONLY,
rustfs_config::DEFAULT_INTERNODE_RPC_MSGPACK_ONLY,
) && rustfs_utils::get_env_bool(
rustfs_config::ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY,
rustfs_config::DEFAULT_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY,
);
INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.store(
if enabled {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_TRUE
} else {
INTERNODE_RPC_MSGPACK_ONLY_CACHE_FALSE
},
Ordering::Release,
);
enabled
}
#[doc(hidden)]
pub fn reset_internode_rpc_msgpack_only_cache() {
INTERNODE_RPC_MSGPACK_ONLY_CACHE.store(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET, Ordering::Release);
INTERNODE_RPC_MSGPACK_PEER_CAPABILITY_CACHE.store(INTERNODE_RPC_MSGPACK_ONLY_CACHE_UNSET, Ordering::Release);
}
/// Consecutive-failure threshold after which an internode peer is marked offline (grpc-optimization
@@ -5784,33 +5757,6 @@ mod tests {
reset_internode_rpc_msgpack_only_cache();
}
#[test]
fn internode_rpc_msgpack_peer_capability_requires_request_and_peer_gate() {
let _guard = INTERNODE_RPC_MSGPACK_ONLY_ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner());
for (requested, peer_capability, expected) in [
(None, None, false),
(Some("true"), None, false),
(None, Some("true"), false),
(Some("true"), Some("false"), false),
(Some("false"), Some("true"), false),
(Some("true"), Some("true"), true),
] {
reset_internode_rpc_msgpack_only_cache();
temp_env::with_vars(
[
(rustfs_config::ENV_INTERNODE_RPC_MSGPACK_ONLY, requested),
(rustfs_config::ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY, peer_capability),
],
|| {
assert_eq!(internode_rpc_msgpack_peer_capability_enabled(), expected);
},
);
}
reset_internode_rpc_msgpack_only_cache();
}
#[tokio::test]
async fn get_channel_for_class_bulk_reuses_control_cache_when_isolation_disabled() {
let _test_guard = TLS02_TEST_LOCK.lock().await;
+1
View File
@@ -15,6 +15,7 @@
pub mod init_time;
pub mod last_minute;
pub mod metrics;
pub mod scan_observation;
pub mod startup;
pub mod usage_candidate;
+40 -1
View File
@@ -13,6 +13,9 @@
// limitations under the License.
use crate::last_minute::{AccElem, LastMinuteLatency};
use crate::scan_observation::{
ScannerMetadataReadGuard, ScannerMetadataReadOutcome, ScannerScanObservation, ScannerScanObservationSnapshot,
};
use crate::usage_candidate::{
ScannerUsageCandidateCount, ScannerUsageCandidateKind, ScannerUsageCandidateOutcome, USAGE_CANDIDATE_COUNTER_SLOTS,
};
@@ -999,6 +1002,7 @@ impl ScannerUsagePublicationObservation {
pub struct Metrics {
startup: crate::startup::ScannerStartupObservation,
scan_observation: ScannerScanObservation,
operations: Vec<AtomicU64>,
latency: Vec<LockedLastMinuteLatency>,
actions: Vec<AtomicU64>,
@@ -1653,6 +1657,9 @@ pub struct ScannerMetricsReport {
pub lifecycle_transition: ScannerLifecycleTransitionSnapshot,
#[serde(default)]
pub usage_freshness: ScannerUsageFreshnessSnapshot,
/// Node-local, process-lifetime work observations. Older nodes have no evidence.
#[serde(default)]
pub scan_observation: ScannerScanObservationSnapshot,
/// Process-lifetime candidate-construction results. Missing on older reports
/// means unobserved, not accepted or healthy. Publication is tracked separately.
#[serde(default)]
@@ -2147,9 +2154,38 @@ impl Metrics {
&self.startup
}
pub fn scan_observation(&self) -> &ScannerScanObservation {
&self.scan_observation
}
/// Start after scanner node admission, before polling the disk read future.
/// Disk-permit and RPC waits remain included; decoding is excluded. Dropping
/// an unfinished guard observes cancellation, not physical I/O completion.
pub fn start_scanner_metadata_read(&self) -> ScannerMetadataReadGuard<'_> {
self.scan_observation.start_metadata_read();
ScannerMetadataReadGuard {
metrics: self,
started: std::time::Instant::now(),
outcome: ScannerMetadataReadOutcome::Cancelled,
bytes: 0,
}
}
pub(crate) fn finish_scanner_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
let bytes = if outcome == ScannerMetadataReadOutcome::Success {
bytes
} else {
0
};
self.scan_observation.finish_metadata_read(outcome, bytes, elapsed);
self.operations[Metric::ReadMetadata as usize].fetch_add(1, Ordering::Relaxed);
self.latency[Metric::ReadMetadata as usize].add_size(elapsed, bytes);
}
pub fn new() -> Self {
Self {
startup: crate::startup::ScannerStartupObservation::default(),
scan_observation: ScannerScanObservation::default(),
operations: (0..Metric::Last as usize).map(|_| AtomicU64::new(0)).collect(),
// Each slot gets its own fresh LockedLastMinuteLatency so that
// different metrics never accidentally share state.
@@ -2755,6 +2791,8 @@ impl Metrics {
/// walked the whole namespace, so an operator can tell why scoped
/// convergence is not happening.
pub fn record_scanner_scope_fallback(&self, reason: impl Into<String>) {
let reason = reason.into();
self.scan_observation.record_scope_fallback(&reason);
self.scanner_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
self.scanner_scope_fallback_last_unix_secs
.store(unix_now_secs(), Ordering::Relaxed);
@@ -2762,7 +2800,7 @@ impl Metrics {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
*last_reason = reason.into();
*last_reason = reason;
}
pub fn record_scanner_usage_durable_success(&self) {
@@ -3897,6 +3935,7 @@ impl Metrics {
failed: self.scanner_transition_failed.load(Ordering::Relaxed),
};
m.usage_candidates = self.usage_candidate_counts();
m.scan_observation = self.scan_observation.snapshot();
let usage_save_result = ScannerUsageSaveResult::from_code(self.scanner_usage_last_save_result.load(Ordering::Relaxed));
m.usage_freshness = ScannerUsageFreshnessSnapshot {
dirty_pending_buckets: self.scanner_dirty_usage_pending_buckets.load(Ordering::Relaxed),
@@ -0,0 +1,675 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Node-local, process-lifetime observations of scanner work. Counters are
//! independently loaded, so concurrent reports need not balance exactly.
//! These observations never grant cache reuse or publication authority.
use crate::metrics::Metrics;
use serde::{Deserialize, Serialize};
use std::{
collections::BTreeMap,
sync::atomic::{AtomicU64, Ordering},
time::{Duration, Instant},
};
pub const PREFIX_REUSE_BLOCKERS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
const SCOPE_FALLBACK_REASONS: [&str; 6] = [
"dirty_snapshot_incomplete",
"dirty_generation_exhausted",
"dirty_bucket_overflow",
"no_listed_dirty_buckets",
"baseline_unavailable",
"peer_scope_unavailable",
];
/// A storage read result, independent of later metadata decoding or ILM work.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerMetadataReadOutcome {
Success,
NotFound,
Error,
Cancelled,
}
/// The existing path that actually copied a cached subtree instead of walking it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseKind {
CleanPrefix,
CoveredNamespace,
SampledNamespace,
ClassifiedNamespace,
}
impl ScannerCacheReuseKind {
const ALL: [Self; 4] = [
Self::CleanPrefix,
Self::CoveredNamespace,
Self::SampledNamespace,
Self::ClassifiedNamespace,
];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::SampledNamespace => "sampled_namespace",
Self::ClassifiedNamespace => "classified_namespace",
}
}
}
/// Completed bucket-level reuse work, independent of durable publication.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBucketReuseKind {
ScopedColdBucket,
CurrentRoot,
}
impl ScannerBucketReuseKind {
const ALL: [Self; 2] = [Self::ScopedColdBucket, Self::CurrentRoot];
fn label(self) -> &'static str {
match self {
Self::ScopedColdBucket => "scoped_cold_bucket",
Self::CurrentRoot => "current_root",
}
}
}
/// A production folder gate, not a diagnostic preflight.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseGate {
CleanPrefix,
CoveredNamespace,
NamespaceSampling,
}
impl ScannerCacheReuseGate {
const ALL: [Self; 3] = [Self::CleanPrefix, Self::CoveredNamespace, Self::NamespaceSampling];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::NamespaceSampling => "namespace_sampling",
}
}
}
/// The first existing condition reached by a folder reuse gate.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseDecision {
MissingScope,
NoResumeSweep,
DeepScan,
ForwardSweep,
ActiveReplication,
NotRoot,
CompactedParent,
MissingCachedChild,
InvalidPrefix,
DirtyPrefix,
ErasureHealthRequired,
MissingCoverage,
CachedFailures,
UncompactedChild,
SelectedForWalk,
Eligible,
}
impl ScannerCacheReuseDecision {
const ALL: [Self; 16] = [
Self::MissingScope,
Self::NoResumeSweep,
Self::DeepScan,
Self::ForwardSweep,
Self::ActiveReplication,
Self::NotRoot,
Self::CompactedParent,
Self::MissingCachedChild,
Self::InvalidPrefix,
Self::DirtyPrefix,
Self::ErasureHealthRequired,
Self::MissingCoverage,
Self::CachedFailures,
Self::UncompactedChild,
Self::SelectedForWalk,
Self::Eligible,
];
fn label(self) -> &'static str {
match self {
Self::MissingScope => "missing_scope",
Self::NoResumeSweep => "no_resume_sweep",
Self::DeepScan => "deep_scan",
Self::ForwardSweep => "forward_sweep",
Self::ActiveReplication => "active_replication",
Self::NotRoot => "not_root",
Self::CompactedParent => "compacted_parent",
Self::MissingCachedChild => "missing_cached_child",
Self::InvalidPrefix => "invalid_prefix",
Self::DirtyPrefix => "dirty_prefix",
Self::ErasureHealthRequired => "erasure_health_required",
Self::MissingCoverage => "missing_coverage",
Self::CachedFailures => "cached_failures",
Self::UncompactedChild => "uncompacted_child",
Self::SelectedForWalk => "selected_for_walk",
Self::Eligible => "eligible",
}
}
fn not_evaluated(self) -> bool {
matches!(
self,
Self::MissingScope | Self::NoResumeSweep | Self::DeepScan | Self::ForwardSweep | Self::ActiveReplication
)
}
}
/// Scope admission and end-of-cycle evidence are separate observations. An
/// activated completion proof does not mean this cycle used prefix reuse.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerPrefixReuseStage {
ScopeResolution,
CycleCompletion,
}
/// The first existing baseline validation check that rejected a snapshot.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBaselineRejection {
Missing,
Decode,
Incomplete,
Partial,
Marks,
Epoch,
SourceCount,
PlanMissing,
PlanMismatch,
SourceIndex,
SourceMembership,
SourceDuplicate,
SetIncomplete,
SetTombstone,
SetEpoch,
SetCycle,
SetPlan,
SourceCoverage,
AuthoritativeIdentity,
NotNewer,
}
impl ScannerBaselineRejection {
const ALL: [Self; 20] = [
Self::Missing,
Self::Decode,
Self::Incomplete,
Self::Partial,
Self::Marks,
Self::Epoch,
Self::SourceCount,
Self::PlanMissing,
Self::PlanMismatch,
Self::SourceIndex,
Self::SourceMembership,
Self::SourceDuplicate,
Self::SetIncomplete,
Self::SetTombstone,
Self::SetEpoch,
Self::SetCycle,
Self::SetPlan,
Self::SourceCoverage,
Self::AuthoritativeIdentity,
Self::NotNewer,
];
fn label(self) -> &'static str {
match self {
Self::Missing => "missing",
Self::Decode => "decode",
Self::Incomplete => "incomplete",
Self::Partial => "partial",
Self::Marks => "marks",
Self::Epoch => "epoch",
Self::SourceCount => "source_count",
Self::PlanMissing => "plan_missing",
Self::PlanMismatch => "plan_mismatch",
Self::SourceIndex => "source_index",
Self::SourceMembership => "source_membership",
Self::SourceDuplicate => "source_duplicate",
Self::SetIncomplete => "set_incomplete",
Self::SetTombstone => "set_tombstone",
Self::SetEpoch => "set_epoch",
Self::SetCycle => "set_cycle",
Self::SetPlan => "set_plan",
Self::SourceCoverage => "source_coverage",
Self::AuthoritativeIdentity => "authoritative_identity",
Self::NotNewer => "not_newer",
}
}
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerMetadataReadSnapshot {
pub attempts: u64,
pub succeeded: u64,
pub not_found: u64,
pub errors: u64,
pub cancelled: u64,
/// Bytes returned by successful storage reads, including corrupt metadata.
pub bytes: u64,
/// Elapsed read-future time, including failed and cancelled reads.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseSnapshot {
/// Completed subtree reuse decisions; not cache entries or objects skipped.
pub count: u64,
/// Time spent copying and attaching the reused subtree, excluding listing.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerBucketReuseSnapshot {
/// Cold bucket copies or current roots successfully handed to the collector.
pub count: u64,
/// Cold copy/attachment time or current-root channel-send time; excludes validation.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseDecisionSnapshot {
/// Policy prerequisites that stopped the gate before cache proof checks.
pub not_evaluated: BTreeMap<String, u64>,
/// First failed cache/work condition; later conditions are not evaluated.
pub blocked: BTreeMap<String, u64>,
/// Admitted candidates, which may still need EC classification before copying.
pub eligible: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerPrefixReuseSnapshot {
pub evaluated: u64,
pub activated: u64,
/// All failed checks are counted; one evaluation can have several blockers.
pub blockers: BTreeMap<String, u64>,
}
/// Missing on an older node means unobserved, not zero work or healthy reuse.
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerScanObservationSnapshot {
pub metadata_reads: ScannerMetadataReadSnapshot,
pub cache_reuse: BTreeMap<String, ScannerCacheReuseSnapshot>,
pub bucket_reuse: BTreeMap<String, ScannerBucketReuseSnapshot>,
pub reuse_decisions: BTreeMap<String, ScannerCacheReuseDecisionSnapshot>,
/// Clean root children still walked because EC scans also check health.
pub erasure_clean_prefix_walks: u64,
pub prefix_scope_resolution: ScannerPrefixReuseSnapshot,
pub prefix_cycle_completion: ScannerPrefixReuseSnapshot,
pub scope_fallbacks: BTreeMap<String, u64>,
/// Rejections only for cycles with a baseline_unavailable scope fallback.
pub authoritative_baseline_rejections: BTreeMap<String, u64>,
pub observed_baseline_rejections: BTreeMap<String, u64>,
}
#[derive(Default)]
pub struct ScannerScanObservation {
metadata: [AtomicU64; 7],
cache_count: [AtomicU64; 4],
cache_nanos: [AtomicU64; 4],
bucket_count: [AtomicU64; 2],
bucket_nanos: [AtomicU64; 2],
reuse_decisions: [[AtomicU64; ScannerCacheReuseDecision::ALL.len()]; 3],
erasure_clean_prefix_walks: AtomicU64,
prefix_evaluated: [AtomicU64; 2],
prefix_activated: [AtomicU64; 2],
prefix_blockers: [[AtomicU64; PREFIX_REUSE_BLOCKERS.len()]; 2],
scope_fallbacks: [AtomicU64; SCOPE_FALLBACK_REASONS.len()],
baseline_rejections: [[AtomicU64; ScannerBaselineRejection::ALL.len()]; 2],
}
fn duration_nanos(duration: Duration) -> u64 {
u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX)
}
fn labelled_counts(labels: impl IntoIterator<Item = &'static str>, counters: &[AtomicU64]) -> BTreeMap<String, u64> {
labels
.into_iter()
.zip(counters)
.map(|(label, count)| (label.to_owned(), count.load(Ordering::Relaxed)))
.collect()
}
impl ScannerScanObservation {
pub(crate) fn start_metadata_read(&self) {
self.metadata[0].fetch_add(1, Ordering::Relaxed);
}
pub(crate) fn finish_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
self.metadata[1 + outcome as usize].fetch_add(1, Ordering::Relaxed);
if outcome == ScannerMetadataReadOutcome::Success {
self.metadata[5].fetch_add(bytes, Ordering::Relaxed);
}
self.metadata[6].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
pub fn record_cache_reuse(&self, kind: ScannerCacheReuseKind, elapsed: Duration) {
self.cache_count[kind as usize].fetch_add(1, Ordering::Relaxed);
self.cache_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records completed reuse work only; it grants no publication authority.
pub fn record_bucket_reuse(&self, kind: ScannerBucketReuseKind, count: u64, elapsed: Duration) {
self.bucket_count[kind as usize].fetch_add(count, Ordering::Relaxed);
self.bucket_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records one first-condition decision without inspecting later prerequisites.
pub fn record_cache_reuse_decision(&self, gate: ScannerCacheReuseGate, decision: ScannerCacheReuseDecision) {
self.reuse_decisions[gate as usize][decision as usize].fetch_add(1, Ordering::Relaxed);
}
pub fn record_erasure_clean_prefix_walk(&self) {
self.erasure_clean_prefix_walks.fetch_add(1, Ordering::Relaxed);
}
pub fn record_prefix_reuse(&self, stage: ScannerPrefixReuseStage, activated: bool, blockers: &[Option<&str>; 8]) {
let stage = stage as usize;
self.prefix_evaluated[stage].fetch_add(1, Ordering::Relaxed);
if activated {
self.prefix_activated[stage].fetch_add(1, Ordering::Relaxed);
}
for blocker in blockers.iter().flatten() {
if let Some(index) = PREFIX_REUSE_BLOCKERS.iter().position(|label| label == blocker) {
self.prefix_blockers[stage][index].fetch_add(1, Ordering::Relaxed);
}
}
}
pub(crate) fn record_scope_fallback(&self, reason: &str) {
if let Some(index) = SCOPE_FALLBACK_REASONS.iter().position(|label| *label == reason) {
self.scope_fallbacks[index].fetch_add(1, Ordering::Relaxed);
}
}
pub fn record_baseline_rejection(&self, authoritative: ScannerBaselineRejection, observed: Option<ScannerBaselineRejection>) {
self.baseline_rejections[0][authoritative as usize].fetch_add(1, Ordering::Relaxed);
if let Some(observed) = observed {
self.baseline_rejections[1][observed as usize].fetch_add(1, Ordering::Relaxed);
}
}
pub fn snapshot(&self) -> ScannerScanObservationSnapshot {
let prefix = |stage: ScannerPrefixReuseStage| ScannerPrefixReuseSnapshot {
evaluated: self.prefix_evaluated[stage as usize].load(Ordering::Relaxed),
activated: self.prefix_activated[stage as usize].load(Ordering::Relaxed),
blockers: labelled_counts(PREFIX_REUSE_BLOCKERS, &self.prefix_blockers[stage as usize]),
};
ScannerScanObservationSnapshot {
metadata_reads: ScannerMetadataReadSnapshot {
attempts: self.metadata[0].load(Ordering::Relaxed),
succeeded: self.metadata[1].load(Ordering::Relaxed),
not_found: self.metadata[2].load(Ordering::Relaxed),
errors: self.metadata[3].load(Ordering::Relaxed),
cancelled: self.metadata[4].load(Ordering::Relaxed),
bytes: self.metadata[5].load(Ordering::Relaxed),
elapsed_nanos: self.metadata[6].load(Ordering::Relaxed),
},
cache_reuse: ScannerCacheReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerCacheReuseSnapshot {
count: self.cache_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.cache_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
bucket_reuse: ScannerBucketReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerBucketReuseSnapshot {
count: self.bucket_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.bucket_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
reuse_decisions: ScannerCacheReuseGate::ALL
.into_iter()
.map(|gate| {
let counts = &self.reuse_decisions[gate as usize];
let decisions = |not_evaluated| {
ScannerCacheReuseDecision::ALL
.into_iter()
.filter(|decision| {
*decision != ScannerCacheReuseDecision::Eligible && decision.not_evaluated() == not_evaluated
})
.map(|decision| (decision.label().to_owned(), counts[decision as usize].load(Ordering::Relaxed)))
.collect()
};
(
gate.label().to_owned(),
ScannerCacheReuseDecisionSnapshot {
not_evaluated: decisions(true),
blocked: decisions(false),
eligible: counts[ScannerCacheReuseDecision::Eligible as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
erasure_clean_prefix_walks: self.erasure_clean_prefix_walks.load(Ordering::Relaxed),
prefix_scope_resolution: prefix(ScannerPrefixReuseStage::ScopeResolution),
prefix_cycle_completion: prefix(ScannerPrefixReuseStage::CycleCompletion),
scope_fallbacks: labelled_counts(SCOPE_FALLBACK_REASONS, &self.scope_fallbacks),
authoritative_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[0],
),
observed_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[1],
),
}
}
}
/// Dropping an unfinished read records cancellation. The guard observes the
/// read future only; it cannot cancel it or change its storage result.
#[must_use]
pub struct ScannerMetadataReadGuard<'a> {
pub(crate) metrics: &'a Metrics,
pub(crate) started: Instant,
pub(crate) outcome: ScannerMetadataReadOutcome,
pub(crate) bytes: u64,
}
impl ScannerMetadataReadGuard<'_> {
pub fn finish(mut self, outcome: ScannerMetadataReadOutcome, bytes: u64) {
self.outcome = outcome;
self.bytes = bytes;
}
}
impl Drop for ScannerMetadataReadGuard<'_> {
fn drop(&mut self) {
self.metrics
.finish_scanner_metadata_read(self.outcome, self.bytes, self.started.elapsed());
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[tokio::test]
async fn metadata_read_reports_storage_results_and_existing_read_metric() {
let metrics = Metrics::new();
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 127);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::NotFound, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Error, 0);
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!(
(reads.attempts, reads.succeeded, reads.not_found, reads.errors, reads.cancelled),
(4, 2, 1, 1, 0)
);
assert_eq!(reads.bytes, 127);
assert_eq!(report.life_time_ops["read_metadata"], 4);
assert_eq!(report.last_minute.actions["read_metadata"].count, 4);
assert_eq!(report.last_minute.actions["read_metadata"].bytes, 127);
}
#[tokio::test]
async fn metadata_read_cancellation_balances_an_actually_polled_future_once() {
let metrics = Arc::new(Metrics::new());
let worker_metrics = Arc::clone(&metrics);
let (started, admitted) = tokio::sync::oneshot::channel();
let worker = tokio::spawn(async move {
let read = worker_metrics.start_scanner_metadata_read();
started.send(()).expect("observe started read");
std::future::pending::<()>().await;
read.finish(ScannerMetadataReadOutcome::Success, 99);
});
tokio::time::timeout(Duration::from_secs(5), admitted)
.await
.expect("read starts within deadline")
.expect("read started");
let in_flight = metrics.scan_observation().snapshot().metadata_reads;
assert_eq!((in_flight.attempts, in_flight.cancelled, in_flight.succeeded), (1, 0, 0));
worker.abort();
assert!(worker.await.expect_err("read task was aborted").is_cancelled());
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!((reads.attempts, reads.cancelled, reads.succeeded, reads.bytes), (1, 1, 0, 0));
assert_eq!(report.life_time_ops["read_metadata"], 1);
assert_eq!(report.last_minute.actions["read_metadata"].count, 1);
assert_eq!(
Duration::from_nanos(reads.elapsed_nanos).as_secs(),
report.last_minute.actions["read_metadata"].acc_time
);
}
#[tokio::test]
async fn observation_report_keeps_stages_reasons_and_copy_costs_separate() {
let metrics = Metrics::new();
let observation = metrics.scan_observation();
observation.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, Duration::from_nanos(321));
observation.record_bucket_reuse(ScannerBucketReuseKind::ScopedColdBucket, 2, Duration::from_nanos(654));
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, Duration::from_nanos(987));
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::MissingScope);
observation
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::ErasureHealthRequired);
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, ScannerCacheReuseDecision::Eligible);
observation.record_erasure_clean_prefix_walk();
observation.record_prefix_reuse(
ScannerPrefixReuseStage::ScopeResolution,
false,
&[
Some("missing_producer_identity"),
Some("missing_durable_journal_replay"),
None,
None,
None,
None,
None,
None,
],
);
observation.record_prefix_reuse(ScannerPrefixReuseStage::CycleCompletion, true, &[None; 8]);
metrics.record_scanner_scope_fallback("baseline_unavailable");
observation.record_baseline_rejection(ScannerBaselineRejection::Epoch, Some(ScannerBaselineRejection::Missing));
let report = metrics.report().await;
let snapshot = &report.scan_observation;
assert_eq!(snapshot.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.cache_reuse["clean_prefix"].elapsed_nanos, 321);
assert_eq!(snapshot.cache_reuse["sampled_namespace"].count, 0);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].count, 2);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].elapsed_nanos, 654);
assert_eq!(snapshot.bucket_reuse["current_root"].count, 1);
assert_eq!(snapshot.bucket_reuse["current_root"].elapsed_nanos, 987);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"], 1);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].blocked["erasure_health_required"], 1);
assert_eq!(snapshot.reuse_decisions["covered_namespace"].eligible, 1);
let mut prior_observations = serde_json::to_value(snapshot).expect("encode observations");
let fields = prior_observations.as_object_mut().expect("observation object");
fields.remove("bucket_reuse");
fields.remove("reuse_decisions");
let prior: ScannerScanObservationSnapshot =
serde_json::from_value(prior_observations).expect("read earlier observations");
assert!(prior.bucket_reuse.is_empty());
assert!(prior.reuse_decisions.is_empty());
assert_eq!(prior.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.erasure_clean_prefix_walks, 1);
assert_eq!(
(snapshot.prefix_scope_resolution.evaluated, snapshot.prefix_scope_resolution.activated),
(1, 0)
);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_producer_identity"], 1);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_durable_journal_replay"], 1);
assert_eq!(
(snapshot.prefix_cycle_completion.evaluated, snapshot.prefix_cycle_completion.activated),
(1, 1)
);
assert_eq!(
snapshot.scope_fallbacks["baseline_unavailable"],
report.usage_freshness.scope_fallback_total
);
assert_eq!(snapshot.authoritative_baseline_rejections["epoch"], 1);
assert_eq!(snapshot.observed_baseline_rejections["missing"], 1);
let mut legacy = serde_json::to_value(&report).expect("encode report");
legacy
.as_object_mut()
.expect("report is an object")
.remove("scan_observation");
let decoded: crate::metrics::ScannerMetricsReport = serde_json::from_value(legacy).expect("read legacy report");
assert!(decoded.scan_observation.cache_reuse.is_empty(), "legacy counters are unobserved");
assert_eq!(decoded.scan_observation.metadata_reads.attempts, 0);
}
}
+26
View File
@@ -206,6 +206,32 @@ pub(crate) struct DataUsageCacheRevisions {
backup: Option<DataUsageCacheRevision>,
}
/// In-memory provenance of the decoded cache and the revisions observed by its loader.
#[derive(Clone, Debug)]
pub(crate) struct DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions,
loaded_from_main: bool,
}
impl DataUsageCacheBaselineRevisions {
pub(crate) fn revisions(&self) -> &DataUsageCacheRevisions {
&self.revisions
}
/// A backup may catch up to an unchanged primary without changing the decoded baseline.
/// Both reads must have decoded that primary; an ETag on a corrupt primary is insufficient.
pub(crate) fn permits_unchanged_baseline_since(&self, initial: &Self) -> bool {
if self.revisions == initial.revisions {
return true;
}
self.loaded_from_main
&& initial.loaded_from_main
&& matches!(&initial.revisions.main, DataUsageCacheRevision::Etag(etag) if !etag.is_empty())
&& self.revisions.main == initial.revisions.main
&& self.revisions.backup.as_ref() == Some(&self.revisions.main)
}
}
pub static DATA_USAGE_BUCKET: LazyLock<String> =
LazyLock::new(|| format!("{RUSTFS_META_BUCKET}{SLASH_SEPARATOR}{BUCKET_META_PREFIX}"));
@@ -70,6 +70,7 @@ struct DataUsageCacheLoadResult {
cache: DataUsageCache,
main_revision: DataUsageCacheRevision,
backup_revision: Option<DataUsageCacheRevision>,
loaded_from_main: bool,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -106,6 +107,18 @@ impl DataUsageCache {
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheRevisions> {
self.load_with_baseline_revisions(store, name)
.await
.map(|baseline| baseline.revisions)
}
/// Loads cache revisions with provenance for unchanged-primary backup reconciliation.
/// Missing or rebuilt caches and a decoded backup do not establish a decoded primary.
pub(crate) async fn load_with_baseline_revisions<S: ScannerObjectIO>(
&mut self,
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheBaselineRevisions> {
let backup_name = format!("{name}.bkp");
let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]);
let loaded = Self::load_cache(store.clone(), name).await?;
@@ -133,7 +146,10 @@ impl DataUsageCache {
let main = loaded.main_revision;
*self = loaded.cache;
Ok(DataUsageCacheRevisions { main, backup })
Ok(DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions { main, backup },
loaded_from_main: loaded.loaded_from_main,
})
}
pub(crate) async fn read_revisions<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheRevisions> {
@@ -156,10 +172,13 @@ impl DataUsageCache {
cache,
revision: Some(main_revision),
} => {
// Legacy-location fallback carries Missing for the primary path.
let loaded_from_main = matches!(&main_revision, DataUsageCacheRevision::Etag(_));
return Ok(DataUsageCacheLoadResult {
cache: *cache,
main_revision,
backup_revision: None,
loaded_from_main,
});
}
DataUsageCacheLoadAttempt::Loaded { revision: None, .. } => {
@@ -197,6 +216,7 @@ impl DataUsageCache {
cache: *cache,
main_revision,
backup_revision: Some(backup_revision),
loaded_from_main: false,
});
}
DataUsageCacheLoadState::Loaded
@@ -220,6 +240,7 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache missing state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
(DataUsageCacheLoadState::Corrupt, DataUsageCacheLoadState::Missing)
@@ -239,6 +260,7 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache corrupt state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
_ => {}
@@ -772,6 +772,102 @@ async fn corrupt_primary_cache_loads_valid_backup() {
assert_eq!(store.puts.load(Ordering::SeqCst), 0);
}
#[test]
fn baseline_revisions_allow_only_a_decoded_unchanged_primary_backup_catchup() {
let etag = |value: &str| DataUsageCacheRevision::Etag(value.to_owned());
let initial = DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions {
main: etag("primary"),
backup: Some(etag("old-backup")),
},
loaded_from_main: true,
};
assert!(initial.permits_unchanged_baseline_since(&initial));
let mut caught_up = initial.clone();
caught_up.revisions.backup = Some(etag("primary"));
assert!(caught_up.permits_unchanged_baseline_since(&initial));
assert!(
!initial.permits_unchanged_baseline_since(&caught_up),
"backup rollback changes the baseline"
);
for backup in [None, Some(DataUsageCacheRevision::Missing)] {
let mut missing_backup = initial.clone();
missing_backup.revisions.backup = backup;
assert!(caught_up.permits_unchanged_baseline_since(&missing_backup));
assert!(!missing_backup.permits_unchanged_baseline_since(&initial));
}
for (initial_from_main, current_from_main) in [(false, true), (true, false), (false, false)] {
let mut baseline = initial.clone();
baseline.loaded_from_main = initial_from_main;
let mut current = caught_up.clone();
current.loaded_from_main = current_from_main;
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"backup fallback grants no primary proof"
);
current.revisions = baseline.revisions.clone();
assert!(
current.permits_unchanged_baseline_since(&baseline),
"exact revisions retain existing admission"
);
}
let mut changed_main = caught_up.clone();
changed_main.revisions.main = etag("new-primary");
changed_main.revisions.backup = Some(etag("new-primary"));
assert!(!changed_main.permits_unchanged_baseline_since(&initial));
let mut changed_backup = caught_up;
changed_backup.revisions.backup = Some(etag("unrelated-backup"));
assert!(!changed_backup.permits_unchanged_baseline_since(&initial));
for main in [DataUsageCacheRevision::Missing, etag("")] {
let mut baseline = initial.clone();
baseline.revisions.main = main.clone();
let mut current = baseline.clone();
current.revisions.backup = Some(main);
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"missing or empty primary ETag is not proof"
);
}
}
#[tokio::test]
async fn baseline_revisions_record_the_actual_decoded_cache_source() {
let mut expected = DataUsageCache::default();
expected.info.name = "decoded".to_owned();
let valid = expected.marshal_msg().expect("encode valid cache");
for (main, backup, from_main, name) in [
(valid.clone(), Some(valid.clone()), true, "decoded"),
(b"corrupt".to_vec(), Some(valid.clone()), false, "decoded"),
(b"corrupt".to_vec(), None, false, ""),
(Vec::new(), None, false, ""),
] {
let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(main), backup));
let mut loaded = DataUsageCache::default();
let baseline = loaded
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
.expect("load actual cache source");
assert_eq!(baseline.loaded_from_main, from_main);
assert_eq!(loaded.info.name, name);
let mut raw_loaded = DataUsageCache::default();
let raw = raw_loaded
.load_with_revisions(store, DATA_USAGE_CACHE_NAME)
.await
.expect("load compatible raw revisions");
assert_eq!(baseline.revisions(), &raw, "raw loader retains its revision contract");
}
let mut legacy_store = CacheReadStore::new(CacheReadBody::Bytes(Vec::new()), None);
legacy_store.legacy = Some(valid);
let mut legacy_cache = DataUsageCache::default();
let legacy = legacy_cache
.load_with_baseline_revisions(Arc::new(legacy_store), "legacy-cache")
.await
.expect("load cache from the legacy location");
assert_eq!(legacy_cache.info.name, "decoded");
assert!(!legacy.loaded_from_main);
assert_eq!(legacy.revisions.main, DataUsageCacheRevision::Missing);
}
#[tokio::test]
async fn corrupt_cache_without_valid_backup_rebuilds_with_cas_revisions() {
for backup in [None, Some(b"also-not-msgpack".to_vec())] {
+8 -9
View File
@@ -248,13 +248,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2;
/// otherwise idle instance whose clean-idle backoff had doubled a 60 s
/// interval), which defeats the fast path it is meant to protect.
///
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] is
/// what protects against a persistently hot bucket driving an unbroken
/// full-scan loop, so it can start small: 5 s, 10 s, 20 s … capped by
/// [`SCANNER_RETRY_MAX_INTERVAL`]. A one-off race recovers in seconds; a
/// genuinely hot bucket still reaches minute-scale backoff within a handful of
/// cycles. Preflight deferrals use the same bounded schedule so a temporarily
/// unavailable peer cannot drive a tight retry loop.
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] protects
/// against a persistently hot bucket driving an unbroken full-scan loop, so
/// retries can start small: 5 s, 10 s, 20 s … Superseded retries cap this
/// schedule at the ordinary cycle interval and preserve an operator-selected
/// nominal cadence. Preflight deferrals retain the generic
/// [`SCANNER_RETRY_MAX_INTERVAL`] cap so a temporarily unavailable peer cannot
/// drive a tight retry loop.
const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5);
const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60);
/// Probe cadence while data movement pauses the scanner. Recovery retries use
@@ -3536,8 +3536,7 @@ where
));
let mut wait_plan =
scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for);
let superseded_retry_interval =
scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features);
let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config);
let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval);
let scope_fallback_retry_interval = scanner_scope_fallback_retry_interval(scope_fallback_backoff, &runtime_config);
let mut convergence_retry_interval = superseded_retry_interval
+7 -10
View File
@@ -192,19 +192,16 @@ impl ScannerRetryBackoff {
pub(super) fn scanner_superseded_retry_interval(
backoff: ScannerRetryBackoff,
runtime_config: &ScannerRuntimeConfig,
maintenance_features: ScannerMaintenanceFeatures,
) -> Option<Duration> {
let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?;
// Preserve an operator-selected duty-cycle floor during convergence.
let cycle_interval = runtime_config.cycle_interval.max(Duration::from_secs(1));
let retry_interval = backoff.retry_interval(cycle_interval)?;
// Preserve an operator-selected nominal cadence during convergence.
if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default {
retry_interval = retry_interval.max(runtime_config.cycle_interval);
return Some(cycle_interval);
}
// A complete but non-authoritative walk must not postpone lifecycle or
// replication evaluation beyond its ordinary maintenance cadence.
if maintenance_features.needs_regular_cycle() {
retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1)));
}
Some(retry_interval)
// A complete but non-authoritative walk must not postpone convergence
// beyond the ordinary cadence, even without lifecycle or replication.
Some(retry_interval.min(cycle_interval))
}
/// Pacing for the cycle after a full-namespace scope fallback while dirty
+73 -38
View File
@@ -8949,7 +8949,7 @@ fn clean_idle_backoff_grows_to_cap() {
}
#[test]
fn superseded_retry_backoff_grows_caps_and_resets_after_convergence() {
fn retry_backoff_grows_caps_and_resets_after_convergence() {
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(backoff.retry_interval(Duration::from_secs(24 * 60 * 60)), None);
@@ -8973,7 +8973,7 @@ fn superseded_retry_backoff_grows_caps_and_resets_after_convergence() {
}
#[test]
fn superseded_retry_backoff_respects_a_faster_configured_cycle() {
fn retry_backoff_respects_a_faster_configured_cycle() {
let mut backoff = ScannerRetryBackoff::default();
backoff.record_retryable_cycle(true);
@@ -8985,7 +8985,7 @@ fn superseded_retry_backoff_respects_a_faster_configured_cycle() {
}
#[test]
fn superseded_retry_backoff_grows_from_the_default_cycle() {
fn retry_backoff_grows_from_the_default_cycle() {
let mut backoff = ScannerRetryBackoff::default();
// The first race after a write burst retries in seconds, not a whole
// cycle, while repeated supersedes still climb toward the cap.
@@ -9005,7 +9005,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()),
scanner_superseded_retry_interval(backoff, &default_config),
Some(SCANNER_RETRY_BASE_INTERVAL),
"the default adaptive cadence must retain fast convergence"
);
@@ -9015,24 +9015,24 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
ScannerRuntimeConfigSource::Config,
ScannerRuntimeConfigSource::ScannerCompatConfig,
] {
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(6 * 60 * 60),
cycle_interval_source: source,
..Default::default()
};
for features in [
ScannerMaintenanceFeatures::default(),
ScannerMaintenanceFeatures {
lifecycle: true,
for cycle_secs in [1, 3, 600, 6 * 60 * 60] {
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
cycle_interval_source: source,
..Default::default()
},
] {
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config, features),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must not be shortened by convergence retries"
);
};
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
for _ in 0..40 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must hold throughout convergence retries"
);
}
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
}
}
}
@@ -9053,7 +9053,7 @@ fn superseded_retry_does_not_delay_required_maintenance() {
for consecutive_cycles in 1..=12 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &config, features),
scanner_superseded_retry_interval(backoff, &config),
Some(config.cycle_interval),
"superseded cycle {consecutive_cycles} must preserve regular maintenance"
);
@@ -9139,19 +9139,50 @@ fn scope_fallback_retry_preserves_explicit_and_fast_cycle_cadence() {
}
#[test]
fn superseded_retry_only_caps_regular_maintenance() {
fn superseded_retry_caps_at_the_default_cycle_and_resets_after_convergence() {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..6 {
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
for expected in [5, 10, 20, 40, 60, 60, 60, 60] {
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(Duration::from_secs(expected)));
}
backoff.consecutive_cycles = u32::MAX;
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
}
#[test]
fn superseded_retry_caps_fast_cycles_without_changing_the_generic_retry_cap() {
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..40 {
backoff.record_retryable_cycle(true);
}
assert_eq!(
scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()),
Some(Duration::from_secs(160)),
backoff.retry_interval(Duration::from_secs(60)),
Some(Duration::from_secs(30 * 60)),
"deferred and transient retries must retain the generic cap"
);
for cycle_secs in [0, 1, 3, 60, 6 * 60 * 60] {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &config),
Some(Duration::from_secs(cycle_secs.max(1))),
);
}
}
#[test]
fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
for features in [
ScannerMaintenanceFeatures {
lifecycle: true,
@@ -9166,28 +9197,23 @@ fn superseded_retry_only_caps_regular_maintenance() {
..Default::default()
},
] {
assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval));
assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None);
assert!(features.requires_full_scan(Some(7), 7, ScannerCycleWakeReason::Timer, true));
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn superseded_maintenance_wait_keeps_convergence_fencing() {
async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let features = ScannerMaintenanceFeatures {
lifecycle: true,
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..12 {
backoff.record_retryable_cycle(true);
}
let interval = scanner_superseded_retry_interval(backoff, &config, features);
let interval = scanner_superseded_retry_interval(backoff, &config);
assert_eq!(interval, Some(config.cycle_interval));
let generations = ScannerCycleObservedGenerations::for_wait(
&config,
@@ -9202,19 +9228,28 @@ async fn superseded_maintenance_wait_keeps_convergence_fencing() {
let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]));
let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]);
let started = tokio::time::Instant::now();
let reason = wait_for_next_scanner_cycle_with_activity(
let mut wait = Box::pin(wait_for_next_scanner_cycle_with_activity(
&ctx,
interval.expect("maintenance must retain its convergence interval"),
interval.expect("superseded retry must retain its convergence interval"),
Some(Duration::from_secs(1)),
&mut seen,
generations,
|| false,
|| std::future::ready(Ok(changed.clone())),
)
.await;
));
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
for _ in 0..3 {
tokio::time::advance(Duration::from_secs(15)).await;
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
}
let reason = wait.await;
assert_eq!(reason, ScannerCycleWakeReason::Timer);
assert_eq!(started.elapsed(), config.cycle_interval);
assert_eq!(seen, Some(changed));
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[test]
@@ -740,6 +740,7 @@ async fn scanner_recovery_intent_startup_rejects_corrupt_pending_record() {
save_config(store.clone(), &path, b"{corrupt".to_vec())
.await
.expect("corrupt intent payload should persist");
crate::scanner_io::tests::wait_for_namespace_commit_tails(store.as_ref()).await;
let restarted = restart_scanner_cycle_store_from(&store).await;
let error = scanner_usage_recovery_intents_for_startup(&CancellationToken::new(), restarted)
+105 -44
View File
@@ -59,6 +59,7 @@ use rustfs_scanner_metrics::metrics::{
CloseDiskGuard, IlmAction, Metric, Metrics, ScannerReplicationRepairKind, ScannerSourceWorkUpdate, ScannerWorkSource,
UpdateCurrentPathFn, current_path_updater, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{ScannerCacheReuseDecision, ScannerCacheReuseGate, ScannerCacheReuseKind};
use rustfs_utils::path::{SLASH_SEPARATOR, path_join_buf};
use sha2::{Digest, Sha256};
use time::OffsetDateTime;
@@ -1060,38 +1061,51 @@ impl FolderScanner {
child_hash: &DataUsageHash,
abandoned_children: &DataUsageHashMap,
) -> bool {
let Some(prefix_scan_scope) = &self.prefix_scan_scope else {
return false;
let decision = if let Some(prefix_scan_scope) = &self.prefix_scan_scope {
if folder.parent.is_some() || folder.name != self.old_cache.info.name {
ScannerCacheReuseDecision::NotRoot
} else if into.compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !abandoned_children.contains(&child_hash.0) {
ScannerCacheReuseDecision::MissingCachedChild
} else if let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
{
if entry.is_empty() || entry.contains('/') {
ScannerCacheReuseDecision::InvalidPrefix
} else if prefix_scan_scope.contains(entry) {
ScannerCacheReuseDecision::DirtyPrefix
} else if self.is_erasure_mode {
// EC scans must still perform independent object-health work.
global_metrics().scan_observation().record_erasure_clean_prefix_walk();
ScannerCacheReuseDecision::ErasureHealthRequired
} else {
ScannerCacheReuseDecision::Eligible
}
} else {
ScannerCacheReuseDecision::InvalidPrefix
}
} else {
ScannerCacheReuseDecision::MissingScope
};
// Erasure-mode usage scans also perform probabilistic object-health
// work. Reusing a clean subtree here would silently suppress that
// independent maintenance path, so prefix reuse is limited to the
// non-erasure data-usage scanner.
if self.is_erasure_mode
|| folder.parent.is_some()
|| folder.name != self.old_cache.info.name
|| into.compacted
|| !abandoned_children.contains(&clone_cache_value(&child_hash.0))
{
return false;
}
let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
else {
return false;
};
!entry.is_empty() && !entry.contains('/') && !prefix_scan_scope.contains(entry)
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, decision);
decision == ScannerCacheReuseDecision::Eligible
}
fn reuse_clean_root_child(&mut self, child: &CachedFolder, child_hash: &DataUsageHash, into: &mut DataUsageEntry) {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
if let Some(updates) = &mut self.usage_updates {
updates.cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
}
into.add_child(child_hash);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, started.elapsed());
}
fn failed_retry_suppressed(&self, path: &str) -> bool {
@@ -1278,14 +1292,60 @@ impl FolderScanner {
}
fn can_reuse_covered_namespace(&self, name: &str, compacted: bool) -> bool {
self.old_cache.info.scan_progress.is_some()
&& !compacted
&& self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
&& self.old_cache.find(name).is_some()
&& !self.resume_subtree_has_failures(name)
let decision = if self.old_cache.info.scan_progress.is_none() {
ScannerCacheReuseDecision::NoResumeSweep
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
{
ScannerCacheReuseDecision::MissingCoverage
} else if self.old_cache.find(name).is_none() {
ScannerCacheReuseDecision::MissingCachedChild
} else if self.resume_subtree_has_failures(name) {
ScannerCacheReuseDecision::CachedFailures
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, decision);
decision == ScannerCacheReuseDecision::Eligible
}
// None preserves work that cannot use sampling; false requests this cycle's rescan.
fn sampled_namespace_reuse(&self, hash: &DataUsageHash, compacted: bool, forward_sweep: bool) -> Option<bool> {
let decision = if self.scan_mode == HealScanMode::Deep {
ScannerCacheReuseDecision::DeepScan
} else if forward_sweep {
ScannerCacheReuseDecision::ForwardSweep
} else if self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
{
ScannerCacheReuseDecision::ActiveReplication
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self.old_cache.is_compacted(hash) {
ScannerCacheReuseDecision::UncompactedChild
} else if hash.mod_(self.old_cache.info.next_cycle as u32, data_usage_update_dir_cycles()) {
ScannerCacheReuseDecision::SelectedForWalk
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::NamespaceSampling, decision);
match decision {
ScannerCacheReuseDecision::Eligible => Some(true),
ScannerCacheReuseDecision::SelectedForWalk => Some(false),
_ => None,
}
}
fn resume_subtree_has_failures(&self, folder: &str) -> bool {
@@ -2077,7 +2137,11 @@ impl FolderScanner {
// The caller stitches this cached child into its
// parent; do not clone a growing child set into the
// unused provisional accumulator as well.
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &this_hash, &folder.parent);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::ClassifiedNamespace, started.elapsed());
self.budget.record_entries_visited(pending_entry_progress);
self.finish_raw_enumeration_parent(&folder.name);
done_folder();
@@ -2664,8 +2728,12 @@ impl FolderScanner {
FolderScanMode::Namespace
};
if child_mode == FolderScanMode::Namespace && reuse_covered {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CoveredNamespace, started.elapsed());
continue;
}
@@ -2705,22 +2773,15 @@ impl FolderScanner {
}
FolderScanSource::Existing => {
// Usage sampling is not proof that a Deep check ran.
if self.scan_mode != HealScanMode::Deep
&& !forward_sweep
&& !self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
&& !into.compacted
&& self.old_cache.is_compacted(&h)
{
let next_cycle = self.old_cache.info.next_cycle as u32;
if !h.mod_(next_cycle, data_usage_update_dir_cycles()) {
if let Some(reuse) = self.sampled_namespace_reuse(&h, into.compacted, forward_sweep) {
if reuse {
if child_mode == FolderScanMode::Namespace {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::SampledNamespace, started.elapsed());
self.record_scan_resume_hint(&folder_item.name);
continue;
}
+191
View File
@@ -3103,6 +3103,112 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Directories));
}
#[tokio::test]
#[serial]
async fn cache_reuse_gates_report_only_the_first_production_condition() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
scanner.old_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
compacted: true,
objects: 2,
..Default::default()
},
);
let hash = hash_path("bucket/child");
let folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
let child = CachedFolder {
name: "bucket/child".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
};
let abandoned = HashSet::from([hash.0.clone()]);
let before = global_metrics().scan_observation().snapshot();
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let missing_scope = global_metrics().scan_observation().snapshot();
assert_eq!(
missing_scope.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"],
before.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"] + 1
);
scanner.prefix_scan_scope = ScannerBucketPrefixScanScope::from_dirty_top_level_entries(HashSet::from(["dirty".to_string()]));
scanner.is_erasure_mode = true;
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let erasure = global_metrics().scan_observation().snapshot();
assert_eq!(
erasure.reuse_decisions["clean_prefix"].blocked["erasure_health_required"],
before.reuse_decisions["clean_prefix"].blocked["erasure_health_required"] + 1
);
assert_eq!(erasure.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_frontier = Some("bucket/child".to_string());
scanner.resume_failed_paths = vec!["bucket/child/failed".to_string()];
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_failed_paths.clear();
assert!(scanner.can_reuse_covered_namespace("bucket/child", false));
let covered = global_metrics().scan_observation().snapshot();
let reasons = &covered.reuse_decisions["covered_namespace"];
let prior = &before.reuse_decisions["covered_namespace"];
assert_eq!(reasons.not_evaluated["no_resume_sweep"], prior.not_evaluated["no_resume_sweep"] + 1);
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["missing_coverage"], prior.blocked["missing_coverage"] + 1);
assert_eq!(reasons.blocked["cached_failures"], prior.blocked["cached_failures"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
with_var(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, Some("16"), || {
scanner.scan_mode = HealScanMode::Deep;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
scanner.scan_mode = HealScanMode::Normal;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
let replication: ReplicationConfig = serde_json::from_value(serde_json::json!({
"config": { "role": "", "rules": [{ "destination": { "bucket": "arn:test" }, "id": "status", "prefix": "", "priority": 1, "status": "Enabled" }] },
"remotes": crate::storage_api::EcstoreBucketTargets::default(),
})).expect("active replication fixture");
scanner.old_cache.info.replication = Some(Arc::new(replication));
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
scanner.old_cache.info.replication = None;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
let rescan_cycle = (0..16)
.find(|cycle| hash.mod_(*cycle, 16))
.expect("hash is visited in one sampling cycle");
scanner.old_cache.info.next_cycle = u64::from(rescan_cycle);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(false));
scanner.old_cache.info.next_cycle = u64::from((rescan_cycle + 1) % 16);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(true));
});
let after = global_metrics().scan_observation().snapshot();
let reasons = &after.reuse_decisions["namespace_sampling"];
let prior = &before.reuse_decisions["namespace_sampling"];
for label in ["deep_scan", "forward_sweep", "active_replication"] {
assert_eq!(
reasons.not_evaluated[label],
prior.not_evaluated[label] + 1,
"first sampling condition {label}"
);
}
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["selected_for_walk"], prior.blocked["selected_for_walk"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
for (kind, reuse) in &after.cache_reuse {
assert_eq!(reuse.count, before.cache_reuse[kind].count, "gate admission alone does not copy {kind}");
}
}
#[tokio::test]
#[serial]
async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entries() {
@@ -3144,12 +3250,21 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
.await
.expect("scoped root scan should finish successfully");
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count + 1);
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks);
assert_eq!(
observed.reuse_decisions["clean_prefix"].eligible,
before.reuse_decisions["clean_prefix"].eligible + 1
);
let clean = scanner
.new_cache
.size_recursive("bucket/clean")
@@ -3167,6 +3282,78 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
assert_eq!((bucket.size, bucket.objects), (17, 3));
}
#[tokio::test]
#[serial]
async fn scoped_cache_observations_distinguish_checkpoint_and_periodic_sampling() {
for covered in [true, false] {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
tokio::fs::create_dir_all(temp_dir.join("bucket/child"))
.await
.expect("create cached namespace");
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
size: 17,
objects: 5,
compacted: true,
..Default::default()
},
);
let kind = if covered {
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
scanner.resume_frontier = Some("bucket/child".to_string());
"covered_namespace"
} else {
let hash = hash_path("bucket/child");
let cycle = (0..data_usage_update_dir_cycles())
.find(|cycle| !hash.mod_(*cycle, data_usage_update_dir_cycles()))
.expect("a cycle samples other work");
scanner.old_cache.info.next_cycle = u64::from(cycle);
"sampled_namespace"
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.await
.expect("cached namespace reuses accounting");
let summary = scanner.new_cache.size_recursive("bucket").expect("complete bucket totals");
assert_eq!((summary.size, summary.objects), (17, 5));
let after = global_metrics().scan_observation().snapshot();
for label in [
"clean_prefix",
"covered_namespace",
"sampled_namespace",
"classified_namespace",
] {
assert_eq!(
after.cache_reuse[label].count,
before.cache_reuse[label].count + u64::from(label == kind),
"reuse reason {label}"
);
}
assert_eq!(after.metadata_reads.attempts, before.metadata_reads.attempts);
let gate = if covered { "covered_namespace" } else { "namespace_sampling" };
assert_eq!(after.reuse_decisions[gate].eligible, before.reuse_decisions[gate].eligible + 1);
}
}
async fn scan_hot_cold_segment_fixture(scoped: bool) -> (DataUsageEntry, Vec<String>) {
let (mut scanner, temp_dir) = build_test_scanner().await;
let visits = metadata_visits::observe(&mut scanner);
@@ -3278,6 +3465,7 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
@@ -3289,6 +3477,9 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
.size_recursive("bucket/clean")
.expect("erasure scan should visit the clean entry");
assert_eq!((clean.size, clean.objects), (0, 0));
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count);
}
#[tokio::test]
@@ -774,6 +774,7 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -790,6 +791,15 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
assert_eq!(budget.progress(), (0, 1), "only the parent consumes namespace quota");
assert_eq!(charged_ops(&root), 4, "list both directories once without re-reading covered metadata");
assert_eq!(scanner.new_cache.find(&folder.name).expect("reused candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
@@ -869,6 +879,7 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -889,6 +900,15 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
"sampling still requires full current classification, without metadata re-read"
);
assert_eq!(scanner.new_cache.find(&folder.name).expect("sampled candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
+133 -46
View File
@@ -40,6 +40,7 @@ use rustfs_lock::{LockError, NamespaceLockGuard};
use rustfs_scanner_metrics::metrics::{
Metric, Metrics, emit_scan_bucket_drive_complete, emit_scan_bucket_drive_partial, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{PREFIX_REUSE_BLOCKERS, ScannerBaselineRejection, ScannerPrefixReuseStage};
use rustfs_utils::path::path_join_buf;
use sha2::{Digest as _, Sha256};
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
@@ -112,16 +113,7 @@ pub(crate) const SCANNER_SEGMENT_ACTIVATION_PROOF_INPUTS: [&str; 9] = [
"durable_dirty_producer_journal",
"ack_generation_guard",
];
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = PREFIX_REUSE_BLOCKERS;
pub type DirtyUsageBuckets = HashMap<String, u64>;
@@ -197,6 +189,7 @@ struct VerifiedRemoteDirtyUsage {
}
struct ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Option<ScannerSegmentReuseActivationPreflight>,
scope: ScannerBucketScanScope,
remote_dirty_usage_acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
distributed_segment_invalidation_evidence: Option<DistributedSegmentInvalidationEvidence>,
@@ -236,6 +229,13 @@ pub(crate) struct ScannerSegmentReuseActivationPreflight {
}
impl ScannerSegmentReuseActivationPreflight {
fn record_observation(&self, stage: ScannerPrefixReuseStage) {
global_metrics().scan_observation().record_prefix_reuse(
stage,
self.scanner_segment_reuse_activated,
&self.fail_closed_blockers,
);
}
#[cfg(test)]
pub(crate) fn fail_closed_blockers(&self) -> impl Iterator<Item = &'static str> + '_ {
self.fail_closed_blockers.iter().filter_map(|blocker| *blocker)
@@ -319,6 +319,7 @@ fn resolve_remote_dirty_usage_scope(
baseline_proof: ScannerCacheBaselineProof<'_>,
) -> ScannerBucketScopeResolutionResult {
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -410,6 +411,7 @@ fn resolve_remote_dirty_usage_scope(
};
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
scope,
remote_dirty_usage_acknowledgements: scoped_acknowledgements,
distributed_segment_invalidation_evidence,
@@ -427,43 +429,81 @@ fn complete_scanner_cache_snapshot_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> Option<DataUsageScanPlanDigest> {
validate_scanner_cache_snapshot_plan_digest(snapshot, proof, expected_converged).ok()
}
fn validate_scanner_cache_snapshot_plan_digest(
snapshot: &DataUsageInfo,
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineRejection> {
// The observational mark disqualifies only the authoritative slot: an
// observed candidate carries it by construction.
let marks_accepted = match expected_converged {
Some(expected) => snapshot.usage_snapshot_converged == Some(expected),
None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational,
};
if !snapshot.is_complete_bucket_usage_snapshot()
|| snapshot.usage_snapshot_partial
|| !marks_accepted
|| snapshot.scanner_epoch != Some(proof.leader_epoch)
|| snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len()
{
return None;
if !snapshot.is_complete_bucket_usage_snapshot() {
return Err(ScannerBaselineRejection::Incomplete);
}
if snapshot.usage_snapshot_partial {
return Err(ScannerBaselineRejection::Partial);
}
if !marks_accepted {
return Err(ScannerBaselineRejection::Marks);
}
if snapshot.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::Epoch);
}
if snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() {
return Err(ScannerBaselineRejection::SourceCount);
}
// Completed maintenance also covers ordinary usage. Keep its exact stored
// proof for cache reuse, and reject mixtures of different set work proofs.
let baseline_plan_digest = DataUsageScanPlanDigest(snapshot.usage_snapshot_set_states.first()?.scan_plan_digest?);
let baseline_plan_digest = DataUsageScanPlanDigest(
snapshot
.usage_snapshot_set_states
.first()
.and_then(|state| state.scan_plan_digest)
.ok_or(ScannerBaselineRejection::PlanMissing)?,
);
if !scanner_plan_digest_is_work_variant(proof.scan_plan_digest, baseline_plan_digest) {
return None;
return Err(ScannerBaselineRejection::PlanMismatch);
}
let mut states = HashSet::with_capacity(snapshot.usage_snapshot_set_states.len());
for state in &snapshot.usage_snapshot_set_states {
let source = DataUsageCacheSource::new(usize::try_from(state.pool_index).ok()?, usize::try_from(state.set_index).ok()?);
if !proof.expected_sources.contains(&source)
|| !states.insert(source)
|| !state.complete
|| state.tombstone
|| state.scanner_epoch != Some(proof.leader_epoch)
|| state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle)
|| state.scan_plan_digest != Some(baseline_plan_digest.0)
{
return None;
let source = DataUsageCacheSource::new(
usize::try_from(state.pool_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
usize::try_from(state.set_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
);
if !proof.expected_sources.contains(&source) {
return Err(ScannerBaselineRejection::SourceMembership);
}
if !states.insert(source) {
return Err(ScannerBaselineRejection::SourceDuplicate);
}
if !state.complete {
return Err(ScannerBaselineRejection::SetIncomplete);
}
if state.tombstone {
return Err(ScannerBaselineRejection::SetTombstone);
}
if state.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::SetEpoch);
}
if state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle) {
return Err(ScannerBaselineRejection::SetCycle);
}
if state.scan_plan_digest != Some(baseline_plan_digest.0) {
return Err(ScannerBaselineRejection::SetPlan);
}
}
(states == *proof.expected_sources).then_some(baseline_plan_digest)
if states != *proof.expected_sources {
return Err(ScannerBaselineRejection::SourceCoverage);
}
Ok(baseline_plan_digest)
}
/// Whether `digest` is `structural` under any work flag: the same bucket
@@ -493,10 +533,35 @@ fn scanner_baseline_pending_buckets(proof: &ScannerCacheBaselineProof<'_>) -> Ve
}
fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option<DataUsageScanPlanDigest> {
let authoritative = serde_json::from_slice::<DataUsageInfo>(proof.authoritative_data?).ok()?;
if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
return Some(validated_digest);
}
validate_scanner_cache_baseline_plan_digest(proof).ok()
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection,
observed: Option<ScannerBaselineRejection>,
}
fn validate_scanner_cache_baseline_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineValidationFailure> {
let authoritative_error = |authoritative| ScannerBaselineValidationFailure {
authoritative,
observed: None,
};
let authoritative_data = proof
.authoritative_data
.ok_or_else(|| authoritative_error(ScannerBaselineRejection::Missing))?;
let authoritative = serde_json::from_slice::<DataUsageInfo>(authoritative_data)
.map_err(|_| authoritative_error(ScannerBaselineRejection::Decode))?;
let authoritative_rejection = match validate_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
Ok(validated_digest) => return Ok(validated_digest),
Err(reason) => reason,
};
let observed_error = |observed| ScannerBaselineValidationFailure {
authoritative: authoritative_rejection,
observed: Some(observed),
};
// A complete but superseded observation may reuse its per-set cache only
// when it was explicitly tied to the durable authoritative baseline. It
@@ -506,14 +571,18 @@ fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<
&& !authoritative.usage_snapshot_observational)
|| crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative);
if !authoritative_has_identity {
return None;
return Err(observed_error(ScannerBaselineRejection::AuthoritativeIdentity));
}
let observed = serde_json::from_slice::<DataUsageInfo>(proof.observed_candidate_data?).ok()?;
let observed_data = proof
.observed_candidate_data
.ok_or_else(|| observed_error(ScannerBaselineRejection::Missing))?;
let observed =
serde_json::from_slice::<DataUsageInfo>(observed_data).map_err(|_| observed_error(ScannerBaselineRejection::Decode))?;
if !observed_data_usage_is_newer(&observed, &authoritative) {
return None;
return Err(observed_error(ScannerBaselineRejection::NotNewer));
}
complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false))
validate_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)).map_err(observed_error)
}
fn scanner_segment_reuse_baseline_producer_evidence(
@@ -658,6 +727,12 @@ impl ScannerScopeFallbackReason {
}
}
#[derive(Clone, Copy)]
struct ScannerScopeFallback {
reason: ScannerScopeFallbackReason,
baseline_rejection: Option<ScannerBaselineValidationFailure>,
}
/// Classify a resolved full-namespace scope against the preconditions
/// `resolve_scanner_bucket_scan_scope` checks, in the same order. Returns
/// `None` when the cycle was required to walk everything or was scoped.
@@ -668,18 +743,24 @@ fn scanner_scope_fallback_reason(
all_buckets: &[BucketInfo],
baseline_proof: ScannerCacheBaselineProof<'_>,
distributed: bool,
) -> Option<ScannerScopeFallbackReason> {
) -> Option<ScannerScopeFallback> {
let fallback = |reason| {
Some(ScannerScopeFallback {
reason,
baseline_rejection: None,
})
};
if !scope_eligible || !resolved_scope.is_default() {
return None;
}
if dirty_usage_snapshot.generation == u64::MAX {
return Some(ScannerScopeFallbackReason::DirtyGenerationExhausted);
return fallback(ScannerScopeFallbackReason::DirtyGenerationExhausted);
}
if !dirty_usage_snapshot.covers_listed_pending {
return Some(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
return fallback(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
}
if dirty_usage_snapshot.buckets.len() > crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES {
return Some(ScannerScopeFallbackReason::DirtyBucketOverflow);
return fallback(ScannerScopeFallbackReason::DirtyBucketOverflow);
}
// Peers may contribute dirty buckets the local snapshot does not hold.
if !distributed {
@@ -689,13 +770,19 @@ fn scanner_scope_fallback_reason(
.keys()
.any(|bucket| listed.contains(bucket.as_str()))
{
return Some(ScannerScopeFallbackReason::NoListedDirtyBuckets);
return fallback(ScannerScopeFallbackReason::NoListedDirtyBuckets);
}
}
if complete_scanner_cache_baseline_plan_digest(baseline_proof).is_none() {
return Some(ScannerScopeFallbackReason::BaselineUnavailable);
if let Err(failure) = validate_scanner_cache_baseline_plan_digest(baseline_proof) {
return Some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::BaselineUnavailable,
baseline_rejection: Some(failure),
});
}
distributed.then_some(ScannerScopeFallbackReason::PeerScopeUnavailable)
distributed.then_some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::PeerScopeUnavailable,
baseline_rejection: None,
})
}
fn scanner_segment_reuse_activation_preflight() -> ScannerSegmentReuseActivationPreflight {
+11 -7
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::*;
/// scanner cache locks and the cache snapshot persist/publish path.
use crate::data_usage_define::clone_cache_value;
use crate::data_usage_define::{DataUsageCacheBaselineRevisions, clone_cache_value};
use crate::{ScannerConfigObjectDelete, ScannerObjectIO};
use rustfs_scanner_metrics::usage_candidate::{
ScannerUsageCandidateKind as CandidateKind, ScannerUsageCandidateOutcome as CandidateOutcome,
@@ -1494,7 +1494,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
store: Arc<SetDisks>,
updates: &mpsc::Sender<DataUsageCache>,
mut cache_snapshot: DataUsageCache,
initial_revisions: Option<&DataUsageCacheRevisions>,
initial_revisions: Option<&DataUsageCacheBaselineRevisions>,
cache_cycle_floor: &AtomicU64,
expected_publication_epoch: u64,
) -> Option<SystemTime> {
@@ -1536,7 +1536,10 @@ pub(super) async fn persist_and_publish_cache_snapshot(
};
let mut persisted = DataUsageCache::default();
let revisions = match persisted.load_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME).await {
let current_baseline = match persisted
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
{
Ok(revisions) => revisions,
Err(err) => {
error!(
@@ -1558,6 +1561,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
return None;
}
};
let revisions = current_baseline.revisions();
let Some(scan_plan_digest) = cache_snapshot.info.scan_plan_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "plan_digest");
return None;
@@ -1621,9 +1625,9 @@ pub(super) async fn persist_and_publish_cache_snapshot(
{
cache_snapshot = persisted;
} else {
// A later execution may have completed while this scan was walking.
// Only replace the cache revision from which this scan started.
if initial_revisions != Some(&revisions) {
// Another execution must not advance the baseline. A backup catching
// up to the same decoded primary changes no baseline content.
if !initial_revisions.is_some_and(|initial| current_baseline.permits_unchanged_baseline_since(initial)) {
warn!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
@@ -1662,7 +1666,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
let done_save = Metrics::time(Metric::SaveUsage);
if let Err(e) = cache_snapshot
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, &revisions, expected_publication_epoch)
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, revisions, expected_publication_epoch)
.await
{
error!(
+48 -3
View File
@@ -14,6 +14,7 @@
use super::*;
/// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::{ScannerBucketReuseKind, ScannerScanObservation};
pub(super) async fn collect_completed_bucket_results(
ctx: CancellationToken,
@@ -142,6 +143,32 @@ pub(super) struct PreparedScopedSetScan {
pub(super) buckets: Vec<BucketInfo>,
pub(super) cache: DataUsageCache,
pub(super) cold_bucket_reuse_proof: Option<ScopedColdBucketReuseProof>,
cold_bucket_reuse_count: u64,
cold_bucket_reuse_elapsed: Duration,
}
impl PreparedScopedSetScan {
pub(super) fn record_cold_bucket_reuse(&self, observation: &ScannerScanObservation) {
observation.record_bucket_reuse(
ScannerBucketReuseKind::ScopedColdBucket,
self.cold_bucket_reuse_count,
self.cold_bucket_reuse_elapsed,
);
}
}
/// Counts only roots delivered to the collector; cancellation or channel failure is not reuse.
pub(super) async fn send_current_cache_root_entry(
bucket_result_tx: &mpsc::Sender<DataUsageEntryInfo>,
root: DataUsageEntryInfo,
cache: &DataUsageCache,
pending_maintenance_work: &AtomicBool,
observation: &ScannerScanObservation,
) -> std::result::Result<(), ScannerError> {
let started = Instant::now();
send_cache_root_entry(bucket_result_tx, root, cache, pending_maintenance_work).await?;
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, started.elapsed());
Ok(())
}
pub(super) struct ScopedColdBucketReuseProof {
@@ -225,6 +252,8 @@ pub(super) fn prepare_scoped_set_scan(
};
cache.replace(DATA_USAGE_ROOT, "", DataUsageEntry::default());
let mut current_bucket_names = HashSet::with_capacity(all_buckets.len());
let mut cold_bucket_reuse_count = 0u64;
let mut cold_bucket_reuse_elapsed = Duration::ZERO;
for bucket in all_buckets {
if !current_bucket_names.insert(bucket.name.as_str()) {
return None;
@@ -233,11 +262,14 @@ pub(super) fn prepare_scoped_set_scan(
cache.info.scan_bucket_value_origins.remove(&bucket.name);
cache.replace(&bucket.name, DATA_USAGE_ROOT, DataUsageEntry::default());
} else {
let started = Instant::now();
cache.copy_with_children(
old_cache,
&rustfs_data_usage::hash_path(&bucket.name),
&Some(rustfs_data_usage::hash_path(DATA_USAGE_ROOT)),
);
cold_bucket_reuse_count = cold_bucket_reuse_count.saturating_add(1);
cold_bucket_reuse_elapsed = cold_bucket_reuse_elapsed.saturating_add(started.elapsed());
}
}
@@ -249,6 +281,8 @@ pub(super) fn prepare_scoped_set_scan(
.collect(),
cache,
cold_bucket_reuse_proof,
cold_bucket_reuse_count,
cold_bucket_reuse_elapsed,
})
}
@@ -425,7 +459,10 @@ impl ScannerIOCache for SetDisks {
.ok_or_else(|| StorageError::other("scanner cache publication is blocked by data movement"))?,
};
let mut old_cache = DataUsageCache::default();
let initial_revisions = match old_cache.load_with_revisions(self.clone(), DATA_USAGE_CACHE_NAME).await {
let initial_revisions = match old_cache
.load_with_baseline_revisions(self.clone(), DATA_USAGE_CACHE_NAME)
.await
{
Ok(revisions) => Some(revisions),
Err(e) => {
warn!(
@@ -555,6 +592,8 @@ impl ScannerIOCache for SetDisks {
})
});
let mut scoped_cache = scoped_scan.map(|mut prepared| {
// The copies are completed work even if this turn later fails publication.
prepared.record_cold_bucket_reuse(global_metrics().scan_observation());
buckets = prepared.buckets;
prepared.cache.info.scan_coverage_digest = Some(bucket_coverage_digest);
prepared.cache
@@ -1558,8 +1597,14 @@ impl ScannerIOCache for SetDisks {
}
record_completed_usage_watch(&usage_watch_completions_clone, source, &bucket.name, bucket_usage_watch.as_ref()).await;
if let Err(e) =
send_cache_root_entry(&bucket_result_tx_clone, *root, &cache, &pending_maintenance_work_clone)
.await
send_current_cache_root_entry(
&bucket_result_tx_clone,
*root,
&cache,
&pending_maintenance_work_clone,
global_metrics().scan_observation(),
)
.await
{
record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await;
error!(
+20 -3
View File
@@ -357,6 +357,7 @@ where
S: ScannerStorage,
{
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -506,7 +507,7 @@ where
distributed,
resolution.baseline_proof,
);
default_result(scoped_scan_scope_from_dirty_buckets(
let scope = scoped_scan_scope_from_dirty_buckets(
resolution.requested_scope,
dirty_buckets,
(!distributed).then_some(resolution.dirty_usage_snapshot.scopes.as_ref()),
@@ -514,7 +515,11 @@ where
segment_reuse_activation_preflight.scanner_segment_reuse_activated,
resolution.all_buckets,
resolution.baseline_proof,
))
);
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
..default_result(scope)
}
}
/// Runs once between the bucket listing and the dirty-usage snapshot, the
@@ -759,6 +764,9 @@ where
},
)
.await;
if let Some(preflight) = scope_resolution.segment_reuse_activation_preflight {
preflight.record_observation(ScannerPrefixReuseStage::ScopeResolution);
}
let remote_dirty_usage_acknowledgements = scope_resolution.remote_dirty_usage_acknowledgements;
let distributed_segment_invalidation_evidence = scope_resolution.distributed_segment_invalidation_evidence;
let scan_scope = scope_resolution.scope;
@@ -770,8 +778,14 @@ where
baseline_proof,
distributed,
);
if let Some(reason) = scope_fallback {
if let Some(fallback) = scope_fallback {
let reason = fallback.reason;
global_metrics().record_scanner_scope_fallback(reason.as_str());
if let Some(failure) = fallback.baseline_rejection {
global_metrics()
.scan_observation()
.record_baseline_rejection(failure.authoritative, failure.observed);
}
tracing::info!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_SET_STATE,
@@ -785,6 +799,7 @@ where
"Scanner cycle walks every bucket because its dirty-bucket scope was not admitted"
);
}
let scope_fallback = scope_fallback.map(|fallback| fallback.reason);
let segment_invalidation_proof = dirty_usage_producer_evidence(&dirty_usage_snapshot).segment_invalidation_proof();
#[cfg(test)]
if let Some(observer) = resolved_scope_observer {
@@ -861,6 +876,7 @@ where
None,
false,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
return Ok(ScannerCycleResult::new(status, dirty_usage_clear)
.with_publication_epoch(publication_epoch)
.with_activity_digest(activity_digest)
@@ -1207,6 +1223,7 @@ where
distributed_segment_invalidation_evidence,
cold_zero_walk_oracle,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
let pending_maintenance_work = pending_maintenance_work_for_cycle(&pending_maintenance_work, &results);
let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire);
let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor);
+15 -2
View File
@@ -15,6 +15,7 @@ use super::*;
use crate::UNKNOWN_TIER;
/// ScannerIODisk implementation for Disk: get_size and the per-disk bucket scan.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::ScannerMetadataReadOutcome;
///
/// Seed [`SizeSummary::tier_stats`] from the cached tier-name list.
@@ -62,8 +63,20 @@ impl ScannerIODisk for Disk {
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let metadata_object_path = item.object_path();
let metadata_result =
crate::workload_admission::with_scanner_metadata_io(self.read_metadata(&item.bucket, &metadata_object_path)).await;
let metadata_result = crate::workload_admission::with_scanner_metadata_io(async {
let read = global_metrics().start_scanner_metadata_read();
let result = self.read_metadata(&item.bucket, &metadata_object_path).await;
let (outcome, bytes) = match &result {
Ok(data) => (ScannerMetadataReadOutcome::Success, u64::try_from(data.len()).unwrap_or(u64::MAX)),
Err(e) if DiskError::is_err_object_not_found(e) || DiskError::is_err_version_not_found(e) => {
(ScannerMetadataReadOutcome::NotFound, 0)
}
Err(_) => (ScannerMetadataReadOutcome::Error, 0),
};
read.finish(outcome, bytes);
result
})
.await;
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let data = match metadata_result {
Ok(data) => data,
+558 -7
View File
@@ -27,10 +27,10 @@ use crate::storage_api::owner::{
};
use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
use crate::{
DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
save_config,
BUCKET_META_PREFIX, DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints,
InstanceContext, PoolEndpoints, ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER,
init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx,
new_disk, path2_bucket_object_with_base_path, save_config,
};
use rustfs_concurrency::{
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
@@ -104,6 +104,7 @@ fn scanner_activity_preflight_defers_a_temporarily_offline_peer() {
#[test]
fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
let before = global_metrics().scan_observation().snapshot();
let preflight = scanner_segment_reuse_activation_preflight();
assert!(preflight.production_activation);
@@ -124,6 +125,9 @@ fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
"distributed_without_peer_invalidation",
]
);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated);
}
#[test]
@@ -2386,6 +2390,254 @@ async fn set_cache_send_failure_logs_receiver_and_snapshot_without_changing_retu
assert_set_cache_context(set_cache_event(&events, "publish_failed"), Some((0, 0)));
}
#[tokio::test]
#[serial]
async fn set_snapshot_accepts_backup_catchup_to_the_same_decoded_primary() {
use crate::data_usage_define::read_config_with_revision;
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read decoded primary bytes");
let primary_body = primary_body.expect("primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode older backup"))
.await
.expect("seed a lagging backup");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture primary with lagging backup");
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up without changing primary");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the reconciled backup");
assert_ne!(initial.revisions(), current.revisions());
assert!(current.permits_unchanged_baseline_since(&initial));
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
candidate.replace(
"photos",
DATA_USAGE_ROOT,
DataUsageEntry {
size: 20,
objects: 1,
..Default::default()
},
);
let cycle = candidate.info.next_cycle;
let cycle_floor = AtomicU64::new(cycle);
let (tx, mut rx) = mpsc::channel(1);
assert!(
persist_and_publish_cache_snapshot(Arc::clone(&set), &tx, candidate, Some(&initial), &cycle_floor, epoch)
.await
.is_some(),
"content-preserving backup catchup must not veto the new snapshot"
);
let published = rx.try_recv().expect("new snapshot reached the collector");
assert_eq!(published.info.next_cycle, cycle);
assert_eq!(published.find("photos").expect("new bucket entry").size, 20);
let (primary, primary_revision) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read new primary");
let (backup, backup_revision) = read_config_with_revision(set, &backup_path).await.expect("read new backup");
assert_eq!(primary, backup, "backup CAS must use the reconciled current revision");
assert_eq!(primary_revision, backup_revision);
let persisted = DataUsageCache::unmarshal(&primary.expect("new primary body")).expect("decode new primary");
assert_eq!(persisted.info.next_cycle, cycle);
assert_eq!(persisted.find("photos").expect("persisted new entry").size, 20);
}
#[tokio::test]
#[serial]
async fn set_snapshot_backup_catchup_keeps_revision_provenance_and_epoch_fences() {
use crate::data_usage_define::read_config_with_revision;
for case in ["different_backup", "different_primary", "corrupt_primary", "changed_epoch"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read original primary");
let primary_body = primary_body.expect("original primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode lagging backup"))
.await
.expect("seed lagging backup");
if case == "corrupt_primary" {
save_config(Arc::clone(&set), &primary_path, b"corrupt-primary".to_vec())
.await
.expect("inject primary corruption before the initial load");
}
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture original baseline source and revisions");
let mut replacement = cache.clone();
replacement.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
match case {
"different_backup" => {
save_config(Arc::clone(&set), &backup_path, replacement.marshal_msg().expect("encode changed backup"))
.await
.expect("backup changes to different contents");
}
"different_primary" => {
let body = replacement.marshal_msg().expect("encode changed primary");
save_config(Arc::clone(&set), &primary_path, body.clone())
.await
.expect("primary advances");
save_config(Arc::clone(&set), &backup_path, body)
.await
.expect("backup follows changed primary");
}
"corrupt_primary" => {
save_config(Arc::clone(&set), &backup_path, b"corrupt-primary".to_vec())
.await
.expect("backup matches the corrupt primary ETag without a decoded primary proof");
}
"changed_epoch" => {
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up before epoch refusal");
}
_ => unreachable!("fixed test cases"),
}
let before_primary = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("current primary");
let before_backup = read_config_with_revision(Arc::clone(&set), &backup_path)
.await
.expect("current backup");
let mut candidate = cache;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let cycle_floor = AtomicU64::new(candidate.info.next_cycle);
let requested_epoch = if case == "changed_epoch" {
epoch.checked_add(1).expect("fixture epoch can advance")
} else {
epoch
};
let (tx, mut rx) = mpsc::channel(1);
let (result, events) = capture_set_cache_logs(persist_and_publish_cache_snapshot(
Arc::clone(&set),
&tx,
candidate,
Some(&initial),
&cycle_floor,
requested_epoch,
))
.await;
assert!(result.is_none(), "{case} must still refuse publication");
assert!(
matches!(rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"{case} forwards no snapshot"
);
let expected_state = if case == "changed_epoch" {
"save_failed"
} else {
"scan_baseline_revision_changed"
};
assert_set_cache_context(set_cache_event(&events, expected_state), Some((0, 0)));
assert_eq!(
read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("protected primary"),
before_primary,
"{case} does not overwrite primary"
);
assert_eq!(
read_config_with_revision(set, &backup_path).await.expect("protected backup"),
before_backup,
"{case} does not overwrite backup"
);
}
}
#[tokio::test]
#[serial]
async fn set_cache_cas_preserves_updates_after_backup_catchup_validation() {
use crate::data_usage_define::read_config_with_revision;
for changed_copy in ["primary", "backup"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read initial primary");
let mut lagging = cache.clone();
lagging.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, lagging.marshal_msg().expect("lagging backup bytes"))
.await
.expect("seed backup lag");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read initial baseline");
save_config(Arc::clone(&set), &backup_path, primary_body.expect("initial primary exists"))
.await
.expect("finish backup catchup");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read reconciled baseline");
assert!(current.permits_unchanged_baseline_since(&initial));
let changed_path = if changed_copy == "primary" {
&primary_path
} else {
&backup_path
};
let mut concurrent = cache.clone();
concurrent.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
let concurrent_body = concurrent.marshal_msg().expect("encode concurrent update");
save_config(Arc::clone(&set), changed_path, concurrent_body.clone())
.await
.expect("copy changes after the baseline validation");
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let result = candidate
.save_with_revisions_for_epoch(Arc::clone(&set), DATA_USAGE_CACHE_NAME, current.revisions(), epoch)
.await;
if changed_copy == "primary" {
assert!(result.is_err(), "primary CAS refuses the intervening update");
} else {
result.expect("primary remains durable when the best-effort backup CAS refuses an update");
}
let (protected, _) = read_config_with_revision(Arc::clone(&set), changed_path)
.await
.expect("read protected concurrent copy");
assert_eq!(
protected,
Some(concurrent_body),
"{changed_copy} CAS must not overwrite the concurrent update"
);
let (primary, _) = read_config_with_revision(set, &primary_path)
.await
.expect("read resulting primary");
let primary = DataUsageCache::unmarshal(&primary.expect("primary exists")).expect("primary remains decodable");
assert_eq!(
primary.info.scan_execution_digest,
Some(DataUsageScanPlanDigest([if changed_copy == "primary" { 9 } else { 3 }; 32])),
);
}
}
#[tokio::test]
#[serial]
async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers() {
@@ -2400,7 +2652,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
.expect("seed legacy set cache");
let mut persisted = DataUsageCache::default();
let initial = persisted
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the shared starting revision");
let mut fresh = legacy.clone();
@@ -2426,7 +2678,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
assert_eq!(published.find("photos").expect("published bucket").size, 20);
assert_eq!(published.info.scan_execution_digest, fresh.info.scan_execution_digest);
let current = persisted
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the current revision for the unidentified execution");
@@ -2717,6 +2969,130 @@ fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provena
);
}
#[test]
fn scoped_scan_baseline_rejections_preserve_the_existing_first_failed_check() {
let source = DataUsageCacheSource::new(1, 2);
let expected_sources = HashSet::from([source]);
let digest = DataUsageScanPlanDigest([9; 32]);
let baseline = complete_usage_baseline(source, digest, 7, 11);
let snapshot: DataUsageInfo = serde_json::from_slice(&baseline).expect("decode complete baseline");
let proof = ScannerCacheBaselineProof {
authoritative_data: Some(&baseline),
observed_candidate_data: None,
expected_sources: &expected_sources,
leader_epoch: 11,
want_cycle: 8,
scan_plan_digest: digest,
};
type RejectionCase = (ScannerBaselineRejection, fn(&mut DataUsageInfo));
let cases: [RejectionCase; 13] = [
(ScannerBaselineRejection::Incomplete, |s| s.usage_snapshot_complete = false),
(ScannerBaselineRejection::Partial, |s| s.usage_snapshot_partial = true),
(ScannerBaselineRejection::Marks, |s| s.usage_snapshot_converged = None),
(ScannerBaselineRejection::Epoch, |s| s.scanner_epoch = Some(10)),
(ScannerBaselineRejection::SourceCount, |s| s.usage_snapshot_set_states.clear()),
(ScannerBaselineRejection::PlanMissing, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = None
}),
(ScannerBaselineRejection::PlanMismatch, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = Some([8; 32])
}),
(ScannerBaselineRejection::SourceMembership, |s| {
s.usage_snapshot_set_states[0].set_index = 3
}),
(ScannerBaselineRejection::SetIncomplete, |s| {
s.usage_snapshot_set_states[0].complete = false
}),
(ScannerBaselineRejection::SetTombstone, |s| {
s.usage_snapshot_set_states[0].tombstone = true
}),
(ScannerBaselineRejection::SetEpoch, |s| {
s.usage_snapshot_set_states[0].scanner_epoch = None
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = Some(9)
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = None
}),
];
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&snapshot, proof, None), Ok(digest));
for (expected, mutate) in cases {
let mut invalid = snapshot.clone();
mutate(&mut invalid);
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None), Err(expected));
assert_eq!(complete_scanner_cache_snapshot_plan_digest(&invalid, proof, None), None);
}
let mut invalid = snapshot.clone();
invalid.usage_snapshot_partial = true;
invalid.scanner_epoch = Some(10);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None),
Err(ScannerBaselineRejection::Partial)
);
let two_sources = HashSet::from([source, DataUsageCacheSource::new(1, 3)]);
let mut two_sets = snapshot.clone();
two_sets
.usage_snapshot_set_states
.push(two_sets.usage_snapshot_set_states[0].clone());
let two_set_proof = ScannerCacheBaselineProof {
expected_sources: &two_sources,
..proof
};
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SourceDuplicate)
);
two_sets.usage_snapshot_set_states[1].set_index = 3;
two_sets.usage_snapshot_set_states[1].scan_plan_digest = Some([8; 32]);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SetPlan)
);
let undecodable = Bytes::from_static(b"not-json");
for (data, expected) in [
(None, ScannerBaselineRejection::Missing),
(Some(&undecodable), ScannerBaselineRejection::Decode),
] {
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
authoritative_data: data,
..proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: expected,
observed: None
})
);
}
let mut wrong_epoch = snapshot;
wrong_epoch.scanner_epoch = Some(10);
let wrong_epoch = Bytes::from(serde_json::to_vec(&wrong_epoch).expect("encode stale epoch baseline"));
let stale_proof = ScannerCacheBaselineProof {
authoritative_data: Some(&wrong_epoch),
..proof
};
assert_eq!(
validate_scanner_cache_baseline_plan_digest(stale_proof),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Missing),
})
);
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
observed_candidate_data: Some(&undecodable),
..stale_proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Decode)
})
);
}
#[test]
fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_baseline() {
let source = DataUsageCacheSource::new(1, 2);
@@ -3150,6 +3526,12 @@ fn distributed_segment_reuse_activation_keeps_remote_dirty_buckets_at_bucket_sco
result.scope.prefix_scope_for("photos").is_some(),
"durable local producer proof may activate local segment reuse after distributed ACK capability evidence"
);
assert!(
result
.segment_reuse_activation_preflight
.expect("actual scope proof is retained for observation")
.scanner_segment_reuse_activated
);
assert!(
result.scope.prefix_scope_for("archive").is_none(),
"remote dirty usage has only bucket-granularity evidence and must not be narrowed to a local prefix"
@@ -3568,6 +3950,10 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
.checked_flatten("stable")
.expect("selected bucket placeholder should exist");
assert_eq!((stable.size, stable.objects), (0, 0));
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
assert!(prepared.cache.find("stable/prefix").is_none());
assert_eq!(prepared.cache.find("dirty").map(|entry| (entry.size, entry.objects)), Some((0, 0)));
assert!(prepared.cache.find("deleted").is_none());
@@ -3579,6 +3965,124 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
assert_eq!(prepared.cache.info.lkg_scan_plan_digest, Some(baseline_digest));
}
#[test]
fn cold_bucket_reuse_observation_rejects_unbound_or_failed_preparation() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
let mut old_cache = complete_set_usage_cache(&[("stable", 10), ("dirty", 20)], baseline_digest);
old_cache.info.scan_bucket_incarnations = test_bucket_incarnations(&["stable", "dirty"]);
let incarnations = old_cache.info.scan_bucket_incarnations.clone();
let buckets = vec![
bucket_info_with_created_time("stable"),
bucket_info_with_created_time("dirty"),
];
let scope = ScannerBucketScanScope {
selected_buckets: Some(Arc::new(HashSet::from(["dirty".to_string()]))),
selected_bucket_prefixes: None,
baseline_scan_plan_digest: Some(baseline_digest),
};
let generation = ScannerSetCacheGeneration {
want_cycle: 8,
leader_epoch: 11,
tier_registry_generation: 13,
source: DataUsageCacheSource::new(1, 2),
scan_plan_digest: DataUsageScanPlanDigest([2; 32]),
};
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
let prepared = prepare_scoped_set_scan(&old_cache, &buckets, &buckets, &scope, generation, Some(&incarnations))
.expect("bound complete cache permits a cold bucket copy");
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 1);
for case in ["incomplete", "duplicate", "nil_incarnation"] {
let mut invalid_cache = old_cache.clone();
let mut invalid_buckets = buckets.clone();
let mut invalid_incarnations = incarnations.clone();
match case {
"incomplete" => invalid_cache.info.snapshot_complete = false,
"duplicate" => invalid_buckets.push(buckets[0].clone()),
"nil_incarnation" => {
invalid_incarnations.insert("stable".to_string(), Uuid::nil());
}
_ => unreachable!("fixed test cases"),
}
let prepared = prepare_scoped_set_scan(
&invalid_cache,
&invalid_buckets,
&invalid_buckets,
&scope,
generation,
Some(&invalid_incarnations),
);
assert!(prepared.is_none(), "{case} must not return a reusable prepared cache");
if let Some(prepared) = prepared {
prepared.record_cold_bucket_reuse(&observation);
}
assert_eq!(
observation.snapshot().bucket_reuse["scoped_cold_bucket"].count,
1,
"{case} does not count completed reuse"
);
}
}
#[tokio::test]
async fn current_root_reuse_counts_only_successful_collector_delivery() {
use rustfs_scanner_metrics::scan_observation::ScannerScanObservation;
let observation = ScannerScanObservation::default();
let cache = DataUsageCache::default();
let pending = AtomicBool::new(false);
let root = || DataUsageEntryInfo {
name: "bucket".to_string(),
parent: DATA_USAGE_ROOT.to_string(),
entry: DataUsageEntry {
size: 10,
objects: 2,
..Default::default()
},
bucket_incarnation: Some(Uuid::from_u128(7)),
tier_registry_generation: None,
value_origin: None,
};
let (sender, mut receiver) = mpsc::channel(1);
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.expect("current root is delivered to the collector");
let delivered = receiver.recv().await.expect("collector received the root");
assert_eq!((delivered.entry.size, delivered.entry.objects), (10, 2));
assert_eq!(observation.snapshot().bucket_reuse["current_root"].count, 1);
drop(receiver);
assert!(
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.is_err()
);
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"closed collector is not completed reuse"
);
let (sender, _receiver) = mpsc::channel(1);
sender.send(root()).await.expect("fill bounded collector channel");
{
let mut send = Box::pin(super::io_cache::send_current_cache_root_entry(
&sender,
root(),
&cache,
&pending,
&observation,
));
assert!(futures::poll!(send.as_mut()).is_pending(), "full collector keeps reuse unfinished");
}
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"dropping unfinished delivery is not reuse"
);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
}
#[test]
fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
@@ -3620,6 +4124,12 @@ fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
)
.expect("matching bucket incarnations should authorize cold bucket reuse");
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
let observed = observation.snapshot();
assert_eq!(observed.bucket_reuse["scoped_cold_bucket"].count, 1);
assert_eq!(observed.bucket_reuse["current_root"].count, 0);
assert!(observed.cache_reuse.values().all(|reuse| reuse.count == 0));
assert_eq!(prepared.buckets.iter().map(|bucket| bucket.name.as_str()).collect::<Vec<_>>(), ["dirty"]);
let stable = prepared
.cache
@@ -4625,12 +5135,36 @@ async fn get_size_treats_missing_metadata_as_skip_file() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let err = disk
.get_size(item)
.get_size(item.clone())
.await
.expect_err("missing metadata should be skipped instead of reported as a scanner failure");
assert!(matches!(err, StorageError::Io(ref io) if io.to_string() == SCANNER_SKIP_FILE_ERROR));
let missing = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(missing.attempts, before.attempts + 1);
assert_eq!(missing.not_found, before.not_found + 1);
assert_eq!(missing.bytes, before.bytes);
#[cfg(unix)]
{
tokio::fs::create_dir_all(&object_dir)
.await
.expect("recreate object directory");
std::os::unix::fs::symlink(STORAGE_FORMAT_FILE, &metadata_path).expect("inject a metadata symlink loop");
let err = disk
.get_size(item)
.await
.expect_err("symlink loop must remain a read failure");
assert!(is_scanner_metadata_transient_error(&err), "read error: {err:?}");
let failed = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(failed.attempts, missing.attempts + 1);
assert_eq!(failed.errors, missing.errors + 1);
assert_eq!(failed.succeeded, before.succeeded);
assert_eq!(failed.bytes, before.bytes);
}
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -4681,6 +5215,7 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
for contents in [b"".as_slice(), b"not-valid-filemeta".as_slice()] {
tokio::fs::write(&metadata_path, contents)
.await
@@ -4692,6 +5227,16 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
assert!(is_scanner_metadata_corrupt_error(&err));
}
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 2);
assert_eq!(
reads.succeeded,
before.succeeded + 2,
"storage reads succeed before decoding rejects corruption"
);
assert_eq!(reads.errors, before.errors);
assert_eq!(reads.bytes, before.bytes + 18);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -4764,11 +5309,17 @@ async fn get_size_counts_delete_markers_separately_from_versions() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let expected_bytes = tokio::fs::metadata(&metadata_path).await.expect("stat stored metadata").len();
let summary = disk.get_size(item).await.expect("scanner should read versioned metadata");
assert_eq!(summary.versions, 2);
assert_eq!(summary.delete_markers, 1);
assert_eq!(summary.total_size, 30);
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 1);
assert_eq!(reads.succeeded, before.succeeded + 1);
assert_eq!(reads.bytes, before.bytes + expected_bytes);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -183,9 +183,11 @@ async fn dirty_entry_removed_during_listing_still_forces_a_full_scan() {
);
assert_eq!(second.buckets_usage["hot-bucket"].objects_count, 2);
assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2);
let freshness = global_metrics().report().await.usage_freshness;
let report = global_metrics().report().await;
let freshness = report.usage_freshness;
assert_eq!(freshness.scope_fallback_total, fallbacks_before + 1);
assert_eq!(freshness.last_scope_fallback_reason, "dirty_snapshot_incomplete");
assert_eq!(report.scan_observation.scope_fallbacks["dirty_snapshot_incomplete"], 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -199,11 +201,53 @@ async fn required_full_scan_is_not_reported_as_a_scope_fallback() {
wait_for_namespace_commit_tails(store.as_ref()).await;
super::io_cycle::set_bucket_listing_window_hook_for_tests(|| record_dirty_usage_bucket("created-after-listing"));
let before = global_metrics().scan_observation().snapshot();
let (result, _, scope) = run_cycle(&store, 2, Some(baseline), true).await;
assert!(scope.is_default());
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert_eq!(result.scope_fallback(), None, "scheduled maintenance walks are not degradations");
assert_eq!(result.completed_scope_fallback(), None);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.scope_fallbacks, before.scope_fallbacks);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn baseline_unavailable_cycle_records_real_scope_and_baseline_rejections_once() {
let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
let _baseline = converged_baseline(&store).await;
put_fixture_object(&store, "hot-bucket", "second").await;
record_dirty_usage_bucket("hot-bucket");
wait_for_namespace_commit_tails(store.as_ref()).await;
let before = global_metrics().scan_observation().snapshot();
let (result, snapshot, scope) = run_cycle(&store, 2, Some(Bytes::from_static(b"not-json")), false).await;
assert!(scope.is_default());
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(result.scope_fallback(), Some(ScannerScopeFallbackReason::BaselineUnavailable));
assert_eq!(snapshot.buckets_usage["hot-bucket"].objects_count, 2);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(
after.scope_fallbacks["baseline_unavailable"],
before.scope_fallbacks["baseline_unavailable"] + 1
);
assert_eq!(
after.authoritative_baseline_rejections["decode"],
before.authoritative_baseline_rejections["decode"] + 1
);
assert_eq!(
after.observed_baseline_rejections, before.observed_baseline_rejections,
"an undecodable authority never admits an observation"
);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated + 1);
assert_eq!(after.prefix_scope_resolution.activated, before.prefix_scope_resolution.activated);
assert_eq!(
after.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"],
before.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"] + 1
);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -1,12 +1,12 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! A bucket whose walk stops early without exhausting the cycle budget (here
//! a sweep resumed under a newer plan, the `mixed_coverage` case of
//! rustfs/rustfs#8494) no longer makes the whole cycle `Incomplete`. The set
//! publishes the bucket's last certified value, the cycle completes with the
//! bucket pending, and the next cycle walks it again. A bucket that never
//! certified keeps the cycle incomplete.
//! A bucket whose walk stops early can publish its last certified value as
//! pending. This fixture tests usage-scan status, not the outer scanner cycle
//! outcome: budget exhaustion may leave complete validated usage while the
//! coordinator records `Partial`. A continuation under a newer plan exercises
//! `mixed_coverage` (rustfs/rustfs#8494). A bucket that never certified keeps
//! the usage scan incomplete.
use super::*;
@@ -38,8 +38,7 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
..Default::default()
},
);
// These cycles persist real bucket checkpoints and source caches. Keep
// CI storage latency inside the production drain window after a count stop.
// Real checkpoint commits need the runtime drain grace after the count budget stops the walk.
budget.use_production_drain_grace_for_tests();
let (updates, mut receiver) = mpsc::channel(1);
let result = tokio::time::timeout(
@@ -49,7 +48,7 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
ScannerCycleRequest {
owner_ctx: ctx.clone(),
ctx: budget.token(),
budget,
budget: Arc::clone(&budget),
updates,
want_cycle: cycle,
leader_epoch: LEADER_EPOCH,
@@ -69,6 +68,13 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
.await
.expect("cycle should finish within the test deadline")
.expect("cycle should succeed");
if options.max_objects.is_some() {
assert_eq!(
budget.reason(),
Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects),
"a count-limited fixture must stop for its object budget"
);
}
let snapshot = receiver.recv().await;
assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot");
(result, snapshot)
@@ -125,9 +131,8 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
assert!(result.clear_verified_usage().is_empty());
assert!(dirty_usage_buckets_for_tests().is_empty());
// Cycle 3: the object budget stops the hot bucket's walk and leaves a
// checkpoint under this cycle's plan. Its certified value is carried, so
// budget exhaustion leaves the bucket pending without blocking the cycle.
// Cycle 3: the budget stops the walk and saves a checkpoint. Usage can
// carry the certified value as pending; the outer coordinator records Partial.
for object in ["p4/third", "p5/third"] {
put_fixture_object(&store, "hot-bucket", object).await;
}
@@ -144,12 +149,38 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
},
)
.await;
let third = third.expect("a carried partial cycle publishes");
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(
result.status,
ScannerCycleStatus::Complete,
"certified carried usage remains complete after a budget stop"
);
let third = third.expect("validated carried usage publishes after the budget stop");
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert!(!third.usage_snapshot_observational);
assert_eq!(third.usage_snapshot_converged, Some(false));
assert_eq!(third.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]);
assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 4);
assert_eq!(third.usage_snapshot_pending_bucket_count, 1);
assert!(third.is_complete_bucket_usage_snapshot());
let carried_hot = &third.buckets_usage["hot-bucket"];
let certified_hot = &second.buckets_usage["hot-bucket"];
assert_eq!(carried_hot.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Carried));
assert!(certified_hot.usage_certification.is_some(), "cycle 2 certified the prior hot value");
assert_eq!(
carried_hot.usage_certification, certified_hot.usage_certification,
"carrying must not mint a new certificate"
);
assert_eq!(carried_hot.objects_count, 4);
assert_eq!(carried_hot.size, certified_hot.size);
assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 1);
assert_eq!(third.buckets_usage["idle-bucket"].objects_count, 1);
assert_eq!(third.objects_total_count, 6);
// Only hot was dirty in this scoped scan; it received no fresh certification or ACK.
assert!(result.clear_verified_usage().is_empty());
assert_eq!(
dirty_usage_buckets_for_tests().keys().cloned().collect::<Vec<_>>(),
vec!["hot-bucket".to_string()],
"the budget-stopped hot bucket retains its dirty responsibility"
);
let set = Arc::clone(&store.pools[0].disk_set[0]);
let mut interrupted = DataUsageCache::default();
interrupted
@@ -60,7 +60,6 @@ Run **before** at the baseline column and **after** at the enabled column, every
| P1 isolation | `RUSTFS_INTERNODE_CHANNEL_ISOLATION` | `false` (default) | `true` |
| P1 bulk pool | `RUSTFS_INTERNODE_BULK_CHANNELS` | unset | `2` |
| P2 msgpack-only | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` + `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` | both `false` (default) | per the convergence runbook |
| P2 per-peer msgpack | `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY` + `RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY` | both `false` (default) | both `true`, fleet confirmation `false`, stable direct/affine peer connections only |
| P3 prewarm | `RUSTFS_INTERNODE_PREWARM` | `false` (default) | `true` |
| P3 offline bypass | `RUSTFS_INTERNODE_OFFLINE_BYPASS` | `false` (default) | `true` |
| P3 reprobe / threshold | `RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS` / `RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD` | defaults | defaults (`DEFAULT_INTERNODE_OFFLINE_REPROBE_SECS`, `DEFAULT_INTERNODE_OFFLINE_FAILURE_THRESHOLD`) |
@@ -1,7 +1,7 @@
# Internode msgpack/JSON convergence runbook
**Use this when:** operating or changing the staged retirement of the JSON compatibility fields on internode gRPC metadata RPCs, deciding whether a fleet may enable `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY`, or adding a new `*_bin` proto field.
**Source of truth:** `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_RPC_MSGPACK_ONLY`, `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED`, `ENV_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY` and their compile-time default-off asserts), `crates/protos/src/node.proto` (`*_bin` fields), `decode_msgpack_or_json` in `rustfs/src/storage/rpc/node_service/disk.rs` (server side) and `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client side), `crates/io-metrics/src/internode_metrics.rs` (counters).
**Source of truth:** `crates/config/src/constants/internode.rs` (`ENV_INTERNODE_RPC_MSGPACK_ONLY`, `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` and their compile-time default-off asserts), `crates/protos/src/node.proto` (`*_bin` fields), `decode_msgpack_or_json` in `rustfs/src/storage/rpc/node_service/disk.rs` (server side) and `crates/ecstore/src/cluster/rpc/remote_disk.rs` (client side), `crates/io-metrics/src/internode_metrics.rs` (counters).
This is a cross-version change and proceeds by observation-gated stages, never in one step.
@@ -16,7 +16,6 @@ Internode RPCs dual-encode each metadata value as a msgpack binary field (`*_bin
| `rustfs_system_network_internode_msgpack_json_decode_total` | `direction`, `message`, `codec` | a msgpack or JSON decode succeeded |
| `rustfs_system_network_internode_msgpack_json_fallback_total` | `direction`, `message` | a decode fell back to JSON because `_bin` was empty |
| `rustfs_system_network_internode_msgpack_json_decode_error_total` | `direction`, `message`, `codec` | either codec failed to decode (`codec` = the failed codec) |
| `rustfs_system_network_internode_msgpack_send_decisions_total` | `decision`, `reason` | an eligible RPC selected `msgpack_only` or `dual_write`; reasons are fixed to `manual_fleet`, `peer_capable`, `peer_unknown`, `peer_revoked`, or `not_requested` |
`direction="request"` is a server decoding a peer's request (`node_service/disk.rs`); `direction="response"` is a client decoding a peer's response (`cluster/rpc/remote_disk.rs`), including the list-level `ReadMultiple` / `BatchReadVersion` fallbacks. `message` is the value name (`FileInfo`, `RawFileInfo`, `ReadMultipleResp`, ...).
@@ -93,55 +92,15 @@ Run the fleet with both flags at their default (`false`) for at least one full o
## Stage 1 — Stop writing JSON (env-gated)
Flag semantics are defined in `crates/config/src/constants/internode.rs`; all three flags default off. `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` requests msgpack-only, and JSON stops only when either fleet confirmation is true or the peer-capability gate is true and the exact current peer boot epoch plus physical connection generation has an authenticated capability proof. Unknown, unsupported, revoked, stale, or reconnecting peers continue dual-write. Decoders keep the JSON fallback in every state.
Flag semantics are defined on `ENV_INTERNODE_RPC_MSGPACK_ONLY` and `ENV_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED` in `crates/config/src/constants/internode.rs`; both default off and only their conjunction stops the JSON write for convergence-ready fields. Decoders keep the JSON fallback in every state.
1. Rehearse `request-only`: `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true`, `..._FLEET_CONFIRMED=false`. Behaviour is unchanged (still dual-write) and safe with old peers; any fallback or decode-error increment still blocks.
2. Canary: set both flags `true` on one node, restart that node only, and watch the fallback and decode-error counters for a soak period with real internode traffic.
3. Fleet: if the counters stay zero, enable both flags fleet-wide with a rolling restart.
4. Rollback: set either flag to `false` (or unset it) and restart; no wire format changed, so rollback is immediate.
An alternative per-peer canary keeps `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY_FLEET_CONFIRMED=false`, sets `RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=true` and `RUSTFS_INTERNODE_RPC_MSGPACK_PEER_CAPABILITY=true`, and confirms `reason="peer_capable"` only for proved peers while every `peer_unknown` or `peer_revoked` decision remains `dual_write`. Enable this mode only when one client HTTP/2 connection has stable affinity to one RustFS peer process; an L7 proxy that routes streams from one connection to different backend versions invalidates the connection-generation proof. A connection-generation change may rebuild one request as dual-write only when the retained local guard proves rejection before the first payload frame; remote status text, a dispatched request, or an ordinary transport error never authorizes that compatibility retry. Disable either the base request flag or peer-capability flag to roll back immediately.
The benchmark driver pins these operator states as dry-run phases (`before`, `request-only`, `canary`, `after`, `rollback`); see [internode-grpc-benchmark-runbook.md](internode-grpc-benchmark-runbook.md).
### Per-peer mixed-version evidence gate
The per-peer gate is not qualified by a successful command transcript alone. Record three ordered four-node measurement phases: `mixed-old-peer` (three candidate nodes and one previous-version node), `all-new` (four candidate nodes), and `rollback-old-peer` (replace one candidate with the previous-version image and verify readback). Every candidate node uses `MSGPACK_ONLY=true`, `MSGPACK_PEER_CAPABILITY=true`, and `MSGPACK_ONLY_FLEET_CONFIRMED=false`; previous-version nodes use all three false. Each phase must run real PUT/GET/metadata traffic and retain the exact candidate/baseline revisions, topology, timestamps, workload result, environment, and raw Prometheus HTTP API JSON.
Capture these PromQL vectors over each non-overlapping phase window and store the complete HTTP API response, not extracted numbers:
```promql
sum by (decision, reason) (increase(rustfs_system_network_internode_msgpack_send_decisions_total[$WINDOW]))
sum(increase(rustfs_system_network_internode_msgpack_json_fallback_total[$WINDOW])) or vector(0)
sum(increase(rustfs_system_network_internode_msgpack_json_decode_error_total[$WINDOW])) or vector(0)
(sum(increase(rustfs_system_network_internode_operation_errors_total{backend="grpc",operation!="grpc_read_version"}[$WINDOW])) or vector(0)) + (sum(increase(rustfs_system_network_internode_operation_classified_errors_total{backend="grpc",operation="grpc_read_version",classification!="remote_response"}[$WINDOW])) or vector(0))
```
Build a schema-1 manifest with `scope="internode-msgpack-peer-canary"`, `source_revision`, `baseline_revision`, `stable_connection_affinity=true`, `max_metric_lag_secs`, and the three phase records. Each phase records integer `start_unix`/`end_unix`, exact topology counts plus a `node1..node4` revision map, the new/old node environment maps, the exact four PromQL strings with a window equal to the phase duration, a relative workload JSON path, and relative paths for `decisions`, `fallback`, `decode_errors`, and `rpc_errors`. The workload artifact records `status`, a non-zero integer `requests`, and `rollback_readback` (true only for `rollback-old-peer`). Validate without rewriting raw evidence:
```bash
python3 scripts/check_internode_msgpack_peer_canary.py \
--manifest /path/to/evidence/manifest.json \
--out /path/to/evidence/verification.json
```
The verifier rejects reused/escaping/overwritten artifacts, duplicate or non-exact decision labels, labeled scalar queries, mismatched PromQL windows, wrong per-node revisions, missing peer-capable traffic, missing mixed/rollback dual-write traffic, manual-fleet or not-requested leakage, failed workload/readback, non-zero fallback/decode/gRPC transport-or-protocol errors, stale timestamps, invalid topology, and candidate/baseline identity reuse. The gRPC error query checks every operation except `grpc_read_version` through the total error counter. Read-version uses its classified counter and excludes only `remote_response`, because a successful gRPC exchange can carry an expected application-level disk response; `grpc_status` and `response_decode` still block the gate. TCP/HTTP operation outcomes remain outside this msgpack gRPC compatibility gate and the workload's zero-error contract remains independently enforced. The workload artifact binds its raw benchmark `summary.csv` by relative path and SHA-256; rollback also records equal source/download content hashes. Verifier output includes SHA-256 receipts for every raw query, workload, and summary artifact. Passing this gate is compatibility evidence only; production soak and ABBA performance drift gates remain separate.
Use the orchestration runner to create the isolated Compose project, execute the three phases, preserve a known object across node2 rollback, capture the four raw Prometheus responses, build the manifest, and invoke the verifier. Without `--execute` it prints a redacted plan and makes no external changes. Execution refuses an existing Compose project/volume set and removes its isolated containers and volumes on exit unless `--keep-up` is explicit.
```bash
scripts/run_internode_msgpack_peer_canary.sh \
--candidate-image rustfs/rustfs:candidate \
--baseline-image rustfs/rustfs:previous \
--candidate-revision <40-hex-candidate-commit> \
--baseline-revision <40-hex-previous-commit> \
--prometheus-query-url http://127.0.0.1:9090/api/v1/query \
--out-dir /path/to/evidence \
--execute
```
The runner requires Docker Compose, AWS CLI, curl, Warp through `run_object_batch_bench.sh`, Python, the four-node local cluster compose file, prebuilt candidate/baseline images, and an already reachable Prometheus/OTLP pipeline. It never builds, pulls, or starts observability images implicitly. Execution rejects reused Compose resources or a non-empty evidence directory and performs a Prometheus vector preflight before starting the cluster. The workload summary must contain only successful zero-error rows with parseable request rates; the rollback phase must read byte-identical object data written before the version replacement.
## Stage 2 — Remove the proto JSON fields
Only after Stage 1 has been stable fleet-wide for a full window with the counters still zero.
@@ -154,7 +113,7 @@ Only after Stage 1 has been stable fleet-wide for a full window with the counter
| Stage | Wire format broken? | Rollback |
| --- | --- | --- |
| 0 Observe | no | n/a (metrics only) |
| 1 msgpack-only send | no | unset the base request flag, or unset the selected fleet/peer gate, and restart |
| 1 msgpack-only send | no | unset either flag and restart |
| 2 remove fields | yes | redeploy the prior release; field numbers stay `reserved` |
## Related
+6
View File
@@ -31,6 +31,12 @@ A previously published gauge can remain visible after a statistic becomes unavai
When live requested bytes are unavailable, capture a supported heap profile or allocation/free trace. Preserve attribution to allocation stacks and lifetime; committed bytes and RSS alone cannot establish a leak. Other raw allocator statistics, including page accounting, depend on the allocator build and are not interchangeable with live application bytes.
### Diagnostic allocator and runtime metrics
Build with `--features diagnostic-allocator-live` to count live requested bytes for Rust allocations routed through the binary's MiMalloc `GlobalAlloc` wrapper. Read `rustfs_memory_allocator_rust_global_requested_live_bytes` together with `rustfs_memory_allocator_rust_global_requested_live_bytes_available`. Availability is `1` for this feature on non-Windows builds when the `hotpath` and `hotpath-alloc` combination is inactive. Direct native allocations are outside this counter. The wrapper records successful allocations, zeroed allocations, reallocations, and frees using allocation-free atomic updates. Per-allocation accounting adds overhead, so collect diagnostic and throughput measurements in separate runs.
The memory sampler also exports `rustfs_runtime_worker_threads`. With `RUSTFLAGS="--cfg tokio_unstable"`, it exports serving-runtime blocking-thread total, idle, and active gauges under `rustfs_runtime_blocking_threads_*`, with `rustfs_runtime_blocking_pool_metrics_available = 1`. An ordinary build reports availability `0`. These metrics describe the serving runtime; isolated disk fsync and cleanup runtimes have their own execution ownership. Enable the allocator feature separately when both views are needed. Tokio's `tokio_unstable` configuration uses the stable compiler and opts into Tokio's unstable telemetry API.
### Linux transparent huge pages
Linux RustFS binaries disable transparent huge pages for mimalloc by default. This reduces resident memory when reclaimable allocations occupy only part of a huge page. Check `THP_enabled` in `/proc/<pid>/status` and `AnonHugePages` in `/proc/<pid>/smaps_rollup` to verify the running process. Both values should be zero with the default. The Linux binary also defaults to immediately purging free arena ranges. Set `MIMALLOC_PURGE_DELAY` before startup to override the delay in milliseconds.
@@ -108,6 +108,28 @@ Claim values are used verbatim as policy names (after `claim_prefix`, if any). N
For first-contact testing only, `RUSTFS_IDENTITY_OPENID_ROLE_POLICY=consoleAdmin` grants every login full access; remove it before production.
### Creating access keys for a local user
In the Console, open **Users → select the local user → Access Keys → Create**. An unrestricted root Console session or a local or OIDC administrator with the built-in `consoleAdmin` policy can create a key for an enabled local user. The key uses the target user's permissions and any narrower session policy supplied at creation.
For a custom administrator, grant both actions. The following policy permits delegated creation only for `backup-user`:
```json
{
"Version": "2012-10-17",
"Statement": [
{"Effect": "Allow", "Action": "admin:CreateServiceAccount"},
{
"Effect": "Allow",
"Action": "admin:CreateServiceAccountForUser",
"Condition": {"StringEquals": {"svc:TargetUser": "backup-user"}}
}
]
}
```
Self-service uses the existing `admin:CreateServiceAccount` permission. The built-in `consoleAdmin` policy and custom policies granting `admin:*` cover both actions. This also applies to existing wildcard grants after an upgrade. To exclude delegation from a wildcard grant, add an explicit deny for `admin:CreateServiceAccountForUser`. Deploy the new server version to all nodes before installing policies that name the new action or condition.
## Validation
1. Discovery:
+14
View File
@@ -193,3 +193,17 @@ The 1024 PUT admission limit includes uploading requests and detached owners;
it is an object-count bound, not a memory or byte budget. The older deferred queue
remains optional and disabled by default; its overflow path cannot bypass the
new coordinator or disk limits. No unbounded foreground fallback is introduced.
## Experimental file-sync follower timeout
`RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_ENABLE` enables file-sync grouping and defaults to `false`. `RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_WAIT_MICROS` controls the batch collection window, defaults to `0`, and is capped at `1000` microseconds. `RUSTFS_EXPERIMENTAL_FILE_FDATASYNC_GROUP_COMMIT_FOLLOWER_TIMEOUT_MICROS` controls how long a follower waits before trying to leave a queued batch. It defaults to `0` (disabled) and is capped at `1000` microseconds. Configure these process-lifetime settings before startup.
After its deadline, a follower can atomically take ownership of its queued files and resource unit. It then executes file sync through the existing per-disk and global limits, followed by source-directory fsync in the normal caller. The original batch loses those paths before the follower can publish its data directory. A worker that has already claimed the batch retains exclusive ownership, and the follower waits for that worker's result. The timeout therefore bounds queued batching delay; disk admission and native sync can take longer. Cancellation retains the existing native-work resource lifetime and worker cleanup guarantees.
The worker guard owns batch claim, in-flight accounting and idle publication. Claiming an empty queue publishes the idle state under the same group lock used by enqueue. A request arriving during cleanup can establish successor-worker ownership, and stale idle cleanup checks that ownership before removing the group.
Pending followers use indexed slab slots and a monotonic waiter identity. Detaching a queued follower performs constant-time lookup under the coordinator locks. The identity check protects a claimed request when its old slot is reused by a later batch. Queue capacity remains bounded by the existing file and waiter limits.
A source that disappears before native open fails its own waiter, while healthy requests in that batch still complete file and directory sync. Other file-sync errors retain the existing fail-fast batch behavior, preventing repeated I/O against a failing disk. Admission or native-dispatch failure also fails the batch. File sets and their existing resource units move together into one blocking dispatch, and result senders remain with the async worker so cancellation keeps its established notification contract.
With detailed PUT metrics enabled, a successful queue ownership transfer adds a `follower_timeout` role sample to `rustfs_s3_put_object_rename_fdatasync_group_wait_ms`. Evaluate the experiment against the same source, durability setting, topology, and workload. Include sync failures, follower publication before batch completion, cancellation, and peak resource usage in acceptance.
@@ -898,6 +898,42 @@ done
The `aggregated.scanner` payload preserves the same scanner progress, checkpoint, pacing, source work, maintenance control, lifecycle expiry, and lifecycle transition fields used by the local scanner status, but only for the responding node; `by_host.*.scanner` keeps that node's host view. Compare the per-node artifacts externally to find old active paths, partial checkpoints, pacing pressure, source-level control pressure, or downstream queue admission problems across the deployment.
## Reading Cached Bucket Statistics
Admin usage reads cache the selected scanner baseline for the existing 30-second TTL, then verify every returned bucket against uncached storage-owned incarnation metadata. Identity reads use at most four concurrent operations and a shared two-second deadline after releasing snapshot/refresh locks. Each bucket read linearizes independently against metadata transactions; the response is not an atomic snapshot of the entire namespace. Delayed peer invalidation cannot authorize a predecessor's statistics for a recreated bucket.
Missing, failed, timed-out, nil, mismatched, or legacy absent identities leave the bucket value absent (unknown), rather than returning the predecessor's value or a confirmed zero. Legacy statistics become visible after a scanner snapshot supplies the current identity. Account-info and usage display apply the existing single-process memory overlay before the final identity filter; distributed local counters are not promoted. When a projection omits values or changes an overlay, totals are partial and `usage_snapshot_converged=false`, and per-tier/replication aggregates are unavailable. Verified cold bucket values remain visible. Such response-only projections cannot be persisted, used for quota admission, or acknowledge dirty work.
## Reading Scanner Work Observations
`GET /rustfs/admin/v3/scanner/status` exposes `metrics.scan_observation` for the responding node. Capture timestamped snapshots on every node, including the scanner leader, and compare their deltas over the same workload window. These process-lifetime counters reset on restart; a missing field on an older node means unobserved. They are loaded independently and may briefly disagree while operations finish. This telemetry does not change scan cadence, cached usage reads, quota authority, dirty acknowledgement, or maintenance requirements.
| Field | Meaning |
|---|---|
| `metadata_reads.attempts` | Disk metadata read futures started after scanner node admission. Invalid paths and operations cancelled before starting the read do not count. |
| `metadata_reads.succeeded`, `not_found`, `errors`, `cancelled` | Storage outcomes, including object/version disappearance and dropped read futures. Successful storage reads can still contain corrupt metadata; decoding and lifecycle failures do not become read errors. Cancellation observes the future, not proof that physical I/O stopped. |
| `metadata_reads.bytes` | Total returned buffer lengths for successful reads, including inline data and corrupt metadata. Failed reads contribute no known bytes; this is not physical disk traffic. |
| `metadata_reads.elapsed_nanos` | Sum of read-future elapsed time, including disk-permit or RPC waits, failures, and cancellation. Folder pacing, scanner node admission, decoding, and lifecycle work are outside this interval. Concurrent reads overlap, so the sum is not cycle wall time. |
| `cache_reuse.clean_prefix` | Clean root subtrees copied under an admitted prefix scope by the non-erasure scanner. |
| `cache_reuse.covered_namespace` | Namespace subtrees copied from validated checkpoint coverage. |
| `cache_reuse.sampled_namespace` | Cached namespace subtrees copied when the existing periodic sampling selected other work. |
| `cache_reuse.classified_namespace` | EC candidate namespace subtrees copied only after the current listing finished classifying them. Health/classification work still runs. |
| `cache_reuse.*.count`, `elapsed_nanos` | Completed subtree reuse decisions and time spent copying/attaching their cache entries. Counts are not objects skipped or a cache hit ratio; times exclude classification/listing and are not total cache maintenance cost. |
| `bucket_reuse.scoped_cold_bucket` | Whole cold buckets copied and attached to a successfully prepared scoped set cache. Count is copied buckets, not plans selected; later turn publication can still fail. Time covers copying/attachment only. |
| `bucket_reuse.current_root` | Existing current bucket roots successfully sent to the local collector after watch, namespace lease and publication epoch checks. Time covers channel send only; failure or cancellation contributes no completed reuse. Collector acceptance and durable publication remain separate. |
| `reuse_decisions.*.not_evaluated`, `blocked`, `eligible` | First conditions reached by production clean-prefix, checkpoint-coverage and namespace-sampling gates. Policy exclusions (`missing_scope`, `no_resume_sweep`, `deep_scan`, `forward_sweep`, `active_replication`) precede cache proof checks. `blocked` reports the first failed work/cache condition; later checks are not repeated for diagnostics. `eligible` does not certify a copy: an EC candidate can still require classification. |
| `erasure_clean_prefix_walks` | Clean root children that still require walking because EC scans perform independent object-health work. |
| `prefix_scope_resolution.evaluated`, `activated`, `blockers` | Proof evaluations retained by the final scope resolver, admitted proof gates, and every failed prerequisite. An admitted proof does not guarantee a reusable child exists; actual subtree copies appear in `cache_reuse`. Startup/required full scans, early scope rejection, and discarded peer/ACK resolutions do not count here. |
| `prefix_cycle_completion.evaluated`, `activated`, `blockers` | End-of-cycle proof evaluations for subsequent reuse. An activated completion proof does not mean this cycle reused prefixes or that its later publication/persistence succeeded. One evaluation can increment several blocker counters. |
| `scope_fallbacks` | Per-reason breakdown of the existing `usage_freshness.scope_fallback_total`; required full scans are excluded. |
| `authoritative_baseline_rejections`, `observed_baseline_rejections` | First failed validation checks for `baseline_unavailable` fallback cycles, preserving the authoritative rejection when the observed fallback also fails. Missing/undecodable authority never reaches observed validation; `authoritative_identity` means it could not anchor an observed fallback. |
`freshness` describes the responding node's completed-cycle age, not whether every bucket has incorporated new writes. Read `metrics.usage_freshness` for node-local live pending work, guardrail vetoes, durable success and publication state; read `metrics.usage_publication` for the confirmed snapshot's pending count and publication class/outcome, and `cycle_schedule` for effective retry intervals and consecutive superseded cycles. Missing or unknown publication data does not establish zero pending work. A follower or a single node's empty dirty map cannot establish cluster-wide convergence.
Whole cold bucket copies, current-root reuse and the four subtree categories have different prerequisites and are not interchangeable hit rates. EC clean prefixes retain health walks. Deep or required full work must run, but a completed cache matching that exact work digest may satisfy a retry of the same cycle. A normal workload need not exercise current-root reuse or every subtree category; use controlled tests for positive emission evidence and capture real gate reasons when a VM window has zero copies.
The existing `life_time_ops.read_metadata` and `last_minute.actions.read_metadata` now include finished and cancelled read attempts; the latter reports count, accumulated whole seconds, and returned bytes in its existing window. Use `metadata_reads.elapsed_nanos` for subsecond read costs. Use read-count and byte deltas to assess work growth, and elapsed-time deltas to assess read-future cost. Compare those with reused-subtree deltas, scope fallback reasons, cycle wall time, pacing waits, and source work before attributing freshness delays to metadata I/O or proposing a scan optimization.
## Reading Lifecycle Transition Status
`metrics.lifecycle_transition`:
Generated
+6 -6
View File
@@ -2,11 +2,11 @@
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1793977579,
"narHash": "sha256-WrSODdZFVIs/Hh2NzC0bY+ROByi0bATg8WfePX8dPj8=",
"lastModified": 1794495034,
"narHash": "sha256-E0hYoioqUtM2XmPPvYYpJgdADF5AW9oOoPSEiM9dTHQ=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "4fe25e12eb9f4c744966dd3de5a05246cf098601",
"rev": "2074cc672349f14240fb7ea90d509e0b643e9dd4",
"type": "github"
},
"original": {
@@ -29,11 +29,11 @@
]
},
"locked": {
"lastModified": 1794040562,
"narHash": "sha256-7eEiezSku3lbZPUOXW8we1a7+xd5axkCQdyI3rzOKjo=",
"lastModified": 1794646145,
"narHash": "sha256-rpqDyUM/d5fACYB8W+wKxxO57ICWjjalzb2X25oFFkY=",
"owner": "oxalica",
"repo": "rust-overlay",
"rev": "d3fe74269f2a6652437e21e78eb7e83be81f6982",
"rev": "5b87ee0513c29127a01ad06e9cb362ea5ce91e78",
"type": "github"
},
"original": {
+1
View File
@@ -15,6 +15,7 @@
use std::io::Write;
fn main() -> shadow_rs::SdResult<()> {
println!("cargo:rustc-check-cfg=cfg(tokio_unstable)");
println!("cargo:rerun-if-env-changed=RUSTFS_BUILD_VERSION");
if let Ok(version) = std::env::var("RUSTFS_BUILD_VERSION")
&& !version.is_empty()
+2 -3
View File
@@ -20,7 +20,7 @@ use crate::admin::service::caller_identity::oidc_profile_fields;
use crate::admin::storage_api::bucket::versioning_sys::BucketVersioningSys;
use crate::admin::storage_api::contract::admin::StorageAdminApi;
use crate::admin::storage_api::contract::bucket::{BucketOperations, BucketOptions};
use crate::admin::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached};
use crate::admin::storage_api::data_usage::load_admin_data_usage_with_memory_overlay;
use crate::admin::storage_api::metadata_sys;
use crate::auth::get_condition_values;
use crate::server::ADMIN_PREFIX;
@@ -278,8 +278,7 @@ impl Operation for AccountInfoHandler {
// process-local absolute counters to cluster-wide bucket totals.
// This path never triggers a live full-version listing
// (rustfs/backlog#1306); freshness is owned by the scanner.
let mut data_usage_info = map_data_usage_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
apply_bucket_usage_memory_overlay(&mut data_usage_info).await;
let data_usage_info = map_data_usage_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
account_info.usage_snapshot_converged = data_usage_info.usage_snapshot_converged;
for bucket in buckets.iter() {
+194 -157
View File
@@ -41,7 +41,7 @@ use rustfs_madmin::{
ServiceAccountInfo, TemporaryAccountInfoResp, UpdateServiceAccountReq,
};
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_policy::policy::{Args, DEFAULT_VERSION, Policy};
use rustfs_policy::policy::{Args, DEFAULT_VERSION, Policy, SVC_TARGET_USER};
use rustfs_s3_types::{Body, S3ErrorCode::InvalidRequest};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
use serde::Deserialize;
@@ -139,19 +139,8 @@ fn is_service_account_owner_of(caller: &StoredCredentials, target_parent_user: &
caller_parent == target_parent_user
}
/// GHSA-5354: whether a caller resolving to `owner` may create a service account
/// parented to `target_user`, given the caller's own key (`req_user`) and its
/// effective parent (`req_parent_user`, which equals `req_user` for a top-level
/// user or the parent for a derived credential).
///
/// The `CreateServiceAccountAdminAction` permission gates *whether* a caller may
/// create service accounts, not *for whom*. A non-owner is therefore confined to
/// its own non-root scope; only an owner may target another user — including the root
/// credential, whose existence check is deliberately lenient. Without this a
/// caller holding only that action could pass `target_user = <root access key>`
/// and mint a root-parented service account that authenticates as owner via
/// `prepare_service_account_auth`, a full-takeover privilege escalation. Mirrors
/// `imported_service_account_parent_allowed` on the ImportIam path (GHSA-566f).
/// Scope available with the base create permission. Creating for another local
/// user requires separate authorization in IAM, even when the caller is owner.
fn add_service_account_parent_within_scope(
root_access_key: &str,
target_user: &str,
@@ -332,6 +321,73 @@ pub fn register_service_account_route(r: &mut S3Router<AdminOperation>) -> std::
Ok(())
}
fn map_service_account_create_error(error: rustfs_iam::error::Error) -> S3Error {
match error {
rustfs_iam::error::Error::NoSuchUser(_) => s3_error!(InvalidRequest, "target user not exist"),
rustfs_iam::error::Error::InvalidArgument
| rustfs_iam::error::Error::InvalidAccessKeyLength
| rustfs_iam::error::Error::InvalidSecretKeyLength
| rustfs_iam::error::Error::AccessKeyAlreadyExists => iam_error_to_s3_error(error),
rustfs_iam::error::Error::IAMActionNotAllowed => s3_error!(AccessDenied, "access denied"),
_ => s3_error!(InternalError, "create service account failed"),
}
}
// The Admin operation owns the local write and site notification together.
// A disconnected requester can stop waiting without splitting those effects.
async fn create_and_replicate_service_account<R: Future<Output = S3Result<()>> + Send>(
create: impl Future<Output = S3Result<(StoredCredentials, OffsetDateTime, HashMap<String, serde_json::Value>)>> + Send + 'static,
replication_policy: Option<String>,
replication_expiration: Option<OffsetDateTime>,
replicate: impl FnOnce(SRIAMItem) -> R + Send + 'static,
) -> S3Result<StoredCredentials> {
super::supervise_admin_mutation("service account creation", async move {
let (new_cred, updated_at, replication_claims) = create.await?;
let (replication_session_policy, oidc_service_account_envelope) =
encode_service_account_replication_policy(&replication_claims, replication_policy.as_deref())?;
if let Err(err) = replicate(SRIAMItem {
r#type: "service-account".to_string(),
svc_acc_change: Some(SRSvcAccChange {
create: Some(SRSvcAccCreate {
parent: new_cred.parent_user.clone(),
access_key: new_cred.access_key.clone(),
secret_key: new_cred.secret_key.clone(),
groups: new_cred.groups.clone().unwrap_or_default(),
claims: replication_claims,
session_policy: replication_session_policy,
status: String::new(),
name: new_cred.name.clone().unwrap_or_default(),
description: new_cred.description.clone().unwrap_or_default(),
expiration: replication_expiration,
api_version: Some(SITE_REPL_API_VERSION.to_string()),
}),
oidc_service_account_envelope,
api_version: Some(SITE_REPL_API_VERSION.to_string()),
..Default::default()
}),
updated_at: Some(updated_at),
api_version: Some(SITE_REPL_API_VERSION.to_string()),
..Default::default()
})
.await
{
warn!(
component = LOG_COMPONENT_ADMIN,
subsystem = LOG_SUBSYSTEM_SERVICE_ACCOUNT,
event = EVENT_ADMIN_SERVICE_ACCOUNT_STATE,
access_key = %new_cred.access_key,
action = "create",
result = "site_replication_hook_failed",
error = ?err,
"admin service account state"
);
}
Ok(new_cred)
})
.await
}
pub struct AddServiceAccount {}
#[async_trait::async_trait]
impl Operation for AddServiceAccount {
@@ -376,20 +432,13 @@ impl Operation for AddServiceAccount {
return Err(s3_error!(InvalidArgument, "can't create user with system access key"));
}
let mut target_user = if let Some(u) = create_req.target_user {
u
} else {
cred.access_key.clone()
};
let req_user = cred.access_key.clone();
let mut req_parent_user = cred.access_key.clone();
let req_groups = cred.groups.clone();
let mut req_is_derived_cred = false;
if cred.is_service_account() || cred.is_temp() {
req_parent_user = cred.parent_user.clone();
req_is_derived_cred = true;
let requested_user = create_req.target_user.as_deref().unwrap_or(&cred.access_key);
let derived = cred.is_service_account() || cred.is_temp();
let caller_parent = if derived { &cred.parent_user } else { &cred.access_key };
let self_scoped = requested_user == cred.access_key || requested_user == caller_parent;
let target_user = if self_scoped { caller_parent.as_str() } else { requested_user };
if target_user.is_empty() {
return Err(s3_error!(AccessDenied, "service account requires a parent"));
}
let Ok(iam_store) = crate::admin::runtime_sources::current_ready_iam_handle() else {
@@ -400,48 +449,45 @@ impl Operation for AddServiceAccount {
let mut conditions = get_condition_values(req.headers(), &cred, None, None, req.policy_source_addr());
set_service_account_duration_condition(&mut conditions, create_req.expiration, OffsetDateTime::now_utc());
// This family deliberately calls `is_allowed` directly instead of going
// through `validate_admin_request`, and must keep doing so
// (backlog#1886). The shared helper returns as soon as *any* candidate
// action is allowed — an OR. The checks here are an AND: each one must
// pass, and a later stage additionally needs `owner` for the GHSA-5354
// parent-scope guard and drives `deny_only` dynamically. Replacing these
// with the shared gate would widen authorization.
if !iam_store
.is_allowed(&Args {
account: &cred.access_key,
groups: &cred.groups,
action: Action::AdminAction(AdminAction::CreateServiceAccountAdminAction),
bucket: "",
conditions: &conditions,
is_owner: owner,
object: "",
claims: cred.claims_or_empty(),
deny_only: false, // Always require explicit Allow permission
})
.await
{
return Err(s3_error!(AccessDenied, "access denied"));
}
if target_user != cred.access_key {
let has_user = iam_store.get_user(&target_user).await;
if has_user.is_none() && target_user != sys_cred.access_key {
// The policy and the write must see the same resolved parent, including
// when a derived credential names itself. OPA receives this map too.
conditions.retain(|key, _| !key.eq_ignore_ascii_case(SVC_TARGET_USER));
conditions.insert(SVC_TARGET_USER.to_owned(), vec![target_user.to_owned()]);
if self_scoped {
if !iam_store
.is_allowed(&Args {
account: &cred.access_key,
groups: &cred.groups,
action: Action::AdminAction(AdminAction::CreateServiceAccountAdminAction),
bucket: "",
conditions: &conditions,
is_owner: owner,
object: "",
claims: cred.claims_or_empty(),
deny_only: false,
})
.await
{
return Err(s3_error!(AccessDenied, "access denied"));
}
if !add_service_account_parent_within_scope(
&sys_cred.access_key,
requested_user,
&cred.access_key,
caller_parent,
owner,
) {
return Err(s3_error!(AccessDenied, "service account parent is outside requester scope"));
}
if requested_user != cred.access_key
&& requested_user != sys_cred.access_key
&& iam_store.get_user(requested_user).await.is_none()
{
return Err(s3_error!(InvalidRequest, "target user not exist"));
}
}
let is_svc_acc = target_user == req_user || target_user == req_parent_user;
// GHSA-5354: confine a non-owner caller to its own scope so it cannot mint
// a root-parented (owner) service account. Evaluated on the original
// `target_user` the caller submitted, before the derived-credential rewrite
// to `req_parent_user` below, so the guard is bound to the parent actually
// requested and to the effective parent after that rewrite.
if !add_service_account_parent_within_scope(&sys_cred.access_key, &target_user, &req_user, &req_parent_user, owner) {
return Err(s3_error!(AccessDenied, "service account parent is outside requester scope"));
}
let mut target_groups = None;
let target_groups = if self_scoped { cred.groups.clone() } else { None };
let opts = NewServiceAccountOpts {
access_key: create_req.access_key,
secret_key: create_req.secret_key,
@@ -452,84 +498,39 @@ impl Operation for AddServiceAccount {
..Default::default()
};
if is_svc_acc {
if req_is_derived_cred {
if req_parent_user.is_empty() {
return Err(s3_error!(AccessDenied, "only derived cred can create service account"));
}
target_user = req_parent_user;
}
target_groups = req_groups;
}
let replication_groups = target_groups.clone().unwrap_or_default();
let replication_name = opts.name.clone().unwrap_or_default();
let replication_description = opts.description.clone().unwrap_or_default();
let target_user = target_user.to_owned();
let response_secret = cred.secret_key.clone();
let replication_expiration = opts.expiration;
let create_result = iam_store
.new_service_account_from_caller(&target_user, target_groups, opts, &cred)
.await;
let (new_cred, updated_at, replication_claims) = create_result.map_err(|e| {
debug!(
component = LOG_COMPONENT_ADMIN,
subsystem = LOG_SUBSYSTEM_SERVICE_ACCOUNT,
event = EVENT_ADMIN_SERVICE_ACCOUNT_STATE,
target_user = %target_user,
result = "create_failed",
error = ?e,
"admin service account state"
);
match e {
rustfs_iam::error::Error::InvalidArgument
| rustfs_iam::error::Error::InvalidAccessKeyLength
| rustfs_iam::error::Error::InvalidSecretKeyLength
| rustfs_iam::error::Error::AccessKeyAlreadyExists => iam_error_to_s3_error(e),
rustfs_iam::error::Error::IAMActionNotAllowed => s3_error!(AccessDenied, "access denied"),
err => s3_error!(InternalError, "create service account failed, e: {:?}", err),
}
})?;
let (replication_session_policy, oidc_service_account_envelope) =
encode_service_account_replication_policy(&replication_claims, replication_policy.as_deref())?;
if let Err(err) = site_replication_iam_change_hook(SRIAMItem {
r#type: "service-account".to_string(),
svc_acc_change: Some(SRSvcAccChange {
create: Some(SRSvcAccCreate {
parent: target_user.clone(),
access_key: new_cred.access_key.clone(),
secret_key: new_cred.secret_key.clone(),
groups: replication_groups,
claims: replication_claims,
session_policy: replication_session_policy,
status: String::new(),
name: replication_name,
description: replication_description,
expiration: replication_expiration,
api_version: Some(SITE_REPL_API_VERSION.to_string()),
}),
oidc_service_account_envelope,
api_version: Some(SITE_REPL_API_VERSION.to_string()),
..Default::default()
}),
updated_at: Some(updated_at),
api_version: Some(SITE_REPL_API_VERSION.to_string()),
..Default::default()
})
.await
{
warn!(
component = LOG_COMPONENT_ADMIN,
subsystem = LOG_SUBSYSTEM_SERVICE_ACCOUNT,
event = EVENT_ADMIN_SERVICE_ACCOUNT_STATE,
access_key = %new_cred.access_key,
action = "create",
result = "site_replication_hook_failed",
error = ?err,
"admin service account state"
);
}
let create = async move {
let create_result = if self_scoped {
iam_store
.new_service_account_from_caller(&target_user, target_groups, opts, &cred)
.await
} else {
iam_store
.new_service_account_for_user(&target_user, opts, &cred, &conditions, owner)
.await
};
create_result.map_err(|e| {
debug!(
component = LOG_COMPONENT_ADMIN,
subsystem = LOG_SUBSYSTEM_SERVICE_ACCOUNT,
event = EVENT_ADMIN_SERVICE_ACCOUNT_STATE,
target_user = %target_user,
result = "create_failed",
error = ?e,
"admin service account state"
);
map_service_account_create_error(e)
})
};
let new_cred = create_and_replicate_service_account(
create,
replication_policy,
replication_expiration,
site_replication_iam_change_hook,
)
.await?;
let resp = AddServiceAccountResp {
credentials: Credentials {
@@ -541,7 +542,7 @@ impl Operation for AddServiceAccount {
};
let body = serde_json::to_vec(&resp).map_err(|e| s3_error!(InternalError, "marshal body failed, e: {:?}", e))?;
let (body, content_type) = encode_compatible_admin_payload(req.uri().path(), &cred.secret_key, body)?;
let (body, content_type) = encode_compatible_admin_payload(req.uri().path(), &response_secret, body)?;
let mut header = HeaderMap::new();
header.insert(CONTENT_TYPE, content_type.parse().expect("valid header value"));
@@ -1851,22 +1852,58 @@ mod tests {
assert_eq!(err.message(), Some("service account 'missing' does not exist"));
}
#[tokio::test]
async fn delegated_service_account_replication_survives_request_cancellation() {
use tokio::sync::oneshot;
use tokio::time::{Duration, timeout};
let (started_tx, started_rx) = oneshot::channel();
let (release_tx, release_rx) = oneshot::channel();
let (event_tx, event_rx) = oneshot::channel();
let updated_at = OffsetDateTime::now_utc();
let create = async move {
started_tx.send(()).expect("signal creation");
release_rx.await.expect("release write");
let credentials = StoredCredentials {
access_key: "backup-key".to_owned(),
parent_user: "backup-user".to_owned(),
..Default::default()
};
Ok((credentials, updated_at, HashMap::new()))
};
let waiter = tokio::spawn(create_and_replicate_service_account(create, None, None, |item| async move {
event_tx.send(item).expect("capture site replication event");
Ok(())
}));
let deadline = Duration::from_secs(5);
timeout(deadline, started_rx)
.await
.expect("creation started")
.expect("start signal");
waiter.abort();
assert!(waiter.await.expect_err("request cancelled").is_cancelled());
release_tx.send(()).expect("write still admitted");
let item = timeout(deadline, event_rx)
.await
.expect("replication completed")
.expect("replication event");
assert_eq!(item.updated_at, Some(updated_at));
let created = item.svc_acc_change.and_then(|change| change.create).expect("create event");
assert_eq!(created.access_key, "backup-key");
assert_eq!(created.parent, "backup-user");
}
#[test]
fn add_service_account_maps_iam_create_errors_to_s3_errors() {
let src = include_str!("service_account.rs");
let create_start = src
.find("impl Operation for AddServiceAccount")
.expect("AddServiceAccount operation should exist");
let create_block = &src[create_start..];
let create_end = create_block
.find("if let Err(err) = site_replication_iam_change_hook")
.expect("site replication hook marker should exist");
let create_block = &create_block[..create_end];
use rustfs_iam::error::Error;
assert!(
create_block.contains("AccessKeyAlreadyExists") && create_block.contains("iam_error_to_s3_error(e)"),
"duplicate access keys must map to client S3 errors"
);
for (error, expected) in [
(Error::AccessKeyAlreadyExists, S3ErrorCode::InvalidArgument),
(Error::NoSuchUser("missing".into()), S3ErrorCode::InvalidRequest),
(Error::Io(std::io::Error::other("storage unavailable")), S3ErrorCode::InternalError),
] {
assert_eq!(map_service_account_create_error(error).code(), &expected);
}
}
#[test]
+6 -4
View File
@@ -945,10 +945,6 @@ pub(crate) static ERR_TIER_RESERVED_NAME: AdminErrorRef = AdminErrorRef(|| &ecst
pub(crate) mod data_usage {
use std::sync::Arc;
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_data_usage_from_backend(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
@@ -960,6 +956,12 @@ pub(crate) mod data_usage {
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
}
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
}
}
pub(crate) mod access {
+65 -22
View File
@@ -21,9 +21,7 @@ use super::storage_api::admin_usecase::capacity::{
};
use super::storage_api::admin_usecase::contract::StorageAdminApi;
use super::storage_api::admin_usecase::contract::bucket::{BucketOperations as _, BucketOptions};
use super::storage_api::admin_usecase::data_usage::{
apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached,
};
use super::storage_api::admin_usecase::data_usage::load_admin_data_usage_with_memory_overlay;
use super::storage_api::admin_usecase::{ECStore, EndpointServerPools};
use crate::app::runtime_sources::{
AppContext, current_app_context, current_endpoints_handle, current_object_store_handle_for_context,
@@ -272,7 +270,7 @@ impl DefaultAdminUsecase {
info.buckets_usage.retain(|bucket, _| current_buckets.contains(bucket));
info.bucket_sizes.retain(|bucket, _| current_buckets.contains(bucket));
if info.buckets_usage.is_empty() && !current_buckets.is_empty() {
if info.buckets_usage.is_empty() && !current_buckets.is_empty() && !info.usage_snapshot_admin_projection {
*info = DataUsageInfo::default();
return;
}
@@ -287,19 +285,14 @@ impl DefaultAdminUsecase {
}
}
/// Overlay single-process counts on a persisted snapshot and narrow it
/// for display.
///
/// Whether the snapshot measured anything is decided before the overlay.
/// The overlay clears `usage_snapshot_complete` when it holds counts it
/// cannot reconcile with the snapshot, such as writes into a bucket that
/// has no verified baseline. That makes the totals unconverged, not the
/// scanned buckets unusable; treating it as structural incompleteness
/// blanked usage for every bucket for as long as one such bucket kept
/// receiving writes, because each write restarts its scanner fence.
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
apply_bucket_usage_memory_overlay(&mut info).await;
/// Keep already identity-bound display values without accepting a display
/// projection as a persistable or quota-authoritative snapshot.
fn present_identity_bound_data_usage_snapshot(
mut info: DataUsageInfo,
buckets: impl IntoIterator<Item = String>,
) -> DataUsageInfo {
let snapshot_measured =
info.usage_snapshot_admin_projection || info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
info
}
@@ -345,7 +338,7 @@ impl DefaultAdminUsecase {
/// This path never triggers a live full-version listing
/// (rustfs/backlog#1306); freshness is owned by the scanner.
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
let info = Self::map_data_usage_load_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
let buckets = store
.list_bucket(&BucketOptions {
cached: true,
@@ -354,7 +347,7 @@ impl DefaultAdminUsecase {
})
.await
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
let mut info = Self::present_identity_bound_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name));
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
@@ -979,6 +972,52 @@ mod tests {
assert_eq!(info.buckets_count, 0);
}
#[test]
fn identity_bound_partial_display_keeps_valid_buckets_and_unknown_state() {
use std::collections::HashMap;
let mut projected = DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
usage_snapshot_admin_projection: true,
usage_snapshot_partial: true,
usage_snapshot_converged: Some(false),
buckets_count: 1,
objects_total_count: 7,
objects_total_size: 70,
buckets_usage: HashMap::from([(
"cold".to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 70,
objects_count: 7,
..Default::default()
},
)]),
bucket_sizes: HashMap::from([("cold".to_string(), 70)]),
..Default::default()
};
projected = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
projected,
["cold".to_string(), "recreated".to_string()],
);
assert_eq!(projected.buckets_usage["cold"].objects_count, 7);
assert!(!projected.buckets_usage.contains_key("recreated"));
assert_eq!(projected.objects_total_count, 7);
assert_eq!(projected.usage_snapshot_converged, Some(false));
assert!(!projected.is_complete_bucket_usage_snapshot());
assert!(!projected.is_valid_partial_snapshot());
projected.buckets_usage.clear();
projected.bucket_sizes.clear();
projected.calculate_totals();
projected.buckets_count = 0;
let unknown = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(projected, ["recreated".to_string()]);
assert!(unknown.buckets_usage.is_empty());
assert!(unknown.usage_snapshot_admin_projection);
assert_eq!(unknown.usage_snapshot_converged, Some(false));
assert!(!unknown.is_complete_bucket_usage_snapshot());
assert_eq!(unknown.last_update, Some(std::time::SystemTime::UNIX_EPOCH));
}
/// A write into a bucket without a verified baseline leaves process-local
/// counts the overlay cannot reconcile, so it clears
/// `usage_snapshot_complete`. That must flag the response non-converged,
@@ -1009,11 +1048,15 @@ mod tests {
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
let info = DefaultAdminUsecase::present_data_usage_snapshot(
// Exercise the production overlay, followed by the same identity-bound
// presentation function used by the admin query.
crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut persisted).await;
persisted.usage_snapshot_admin_projection = true;
persisted.usage_snapshot_converged = Some(false);
let info = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
persisted,
[measured_bucket.to_string(), unverified_bucket.to_string()],
)
.await;
);
assert_eq!(
info.buckets_usage
@@ -108,6 +108,17 @@ async fn persist_scanner_snapshot(ecstore: Arc<ECStore>, info: DataUsageInfo) {
rustfs_scanner::scanner::store_data_usage_in_backend(CancellationToken::new(), ecstore, receiver).await;
}
async fn bind_current_bucket_incarnations(ecstore: &ECStore, info: &mut DataUsageInfo) {
for (bucket, usage) in &mut info.buckets_usage {
let incarnation = ecstore
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("load the current incarnation for the covered snapshot");
assert!(!incarnation.is_nil(), "covered snapshot must bind a non-nil bucket identity");
usage.bucket_incarnation = Some(incarnation.to_string());
}
}
#[tokio::test]
#[serial]
async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
@@ -174,6 +185,7 @@ async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
seeded_info.bucket_sizes.insert(overlay_bucket.clone(), 0);
seeded_info.buckets_count = u64::try_from(seeded_info.buckets_usage.len()).expect("test bucket count should fit into u64");
seeded_info.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut seeded_info).await;
persist_scanner_snapshot(ecstore.clone(), seeded_info).await;
record_bucket_object_write_memory(&overlay_bucket, None, 512).await;
@@ -341,6 +353,7 @@ async fn data_usage_endpoint_treats_legacy_partial_stats_as_unknown_until_covere
completed.bucket_sizes.insert(small_bucket.clone(), 57_344);
completed.buckets_count = u64::try_from(completed.buckets_usage.len()).expect("test bucket count should fit into u64");
completed.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut completed).await;
store_data_usage_in_backend(completed, ecstore.clone())
.await
.expect("persist complete covered snapshot");
+9 -5
View File
@@ -49,7 +49,9 @@ use super::storage_api::multipart_usecase::io::{DecryptReader, EncryptReader, Ha
use super::storage_api::multipart_usecase::io::{
EtagAlgorithm, HashReader, WriteEncryption, WritePlan, compression_metadata_value,
};
use super::storage_api::multipart_usecase::object_utils::{to_s3s_etag, to_s3s_xml_etag};
#[cfg(test)]
use super::storage_api::multipart_usecase::object_utils::to_s3s_etag;
use super::storage_api::multipart_usecase::object_utils::to_s3s_xml_etag;
use super::storage_api::multipart_usecase::options::{
copy_src_opts, extract_metadata_from_mime, get_complete_multipart_upload_opts_with_replication_authorization,
get_content_sha256_with_query, get_opts, has_replication_retention_update, namespace_reserved_user_metadata,
@@ -73,7 +75,6 @@ use super::storage_api::multipart_usecase::{
StorageObjectInfo as ObjectInfo, StorageObjectOptions as ObjectOptions, StoragePutObjReader as PutObjReader,
validate_new_object_key,
};
use super::trailer_adapter::trailer_source;
use crate::app::object::{
ConcurrencyManager, ForegroundWriteAdmission, get_concurrency_manager, put_object_body_read_timeout,
reject_non_wildcard_write_if_none_match, reject_oversize_single_upload, reject_veeam_sosapi_mutation,
@@ -83,8 +84,7 @@ use crate::app::object_data_cache::{
invalidate_object_data_cache_before_mutation,
};
use crate::app::object_usecase::{
apply_quota_admission, build_put_like_object_lock_metadata, map_quota_check_outcome, s3s_body_error_to_io,
validate_existing_object_lock_for_write,
apply_quota_admission, map_quota_check_outcome, s3s_body_error_to_io, validate_existing_object_lock_for_write,
};
use crate::app::runtime_sources::{
AppContext, current_app_context, current_object_data_cache_for_context, current_object_store_handle_for_context,
@@ -129,11 +129,15 @@ use uuid::Uuid;
mod abort;
mod complete;
mod create;
mod create_values;
pub(crate) use create_values::{CreateMultipartUploadFields, CreateMultipartUploadResult};
mod list;
#[cfg(test)]
mod test_support;
mod upload_part;
mod upload_part_copy;
mod upload_values;
pub(crate) use upload_values::{UploadPartTarget, UploadPartWriteInput, UploadPartWriteOutput};
#[cfg(test)]
fn merge_part_encryption_metadata(
@@ -535,7 +539,7 @@ mod tests {
.build()
.expect("valid create request DTO");
let error = make_usecase()
.execute_create_multipart_upload(build_request(create, Method::POST))
.execute_create_multipart_upload_from_s3s(build_request(create, Method::POST))
.await
.expect_err("reserved key must be rejected before accessing storage");
assert_eq!(error.code(), &S3ErrorCode::InvalidArgument);
+6 -6
View File
@@ -1026,7 +1026,7 @@ mod tests {
..Default::default()
});
let upload_id = usecase
.execute_create_multipart_upload(create_request)
.execute_create_multipart_upload_from_s3s(create_request)
.await
.expect("authorized REPLICA-only create should succeed")
.output
@@ -1124,7 +1124,7 @@ mod tests {
.headers
.insert("x-amz-checksum-type", HeaderValue::from_static("COMPOSITE"));
let upload_id = usecase
.execute_create_multipart_upload(create)
.execute_create_multipart_upload_from_s3s(create)
.await
.expect("SSE-C create")
.output
@@ -1514,7 +1514,7 @@ mod tests {
.headers
.insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*"));
let first_upload_id = usecase
.execute_create_multipart_upload(initial_create_req)
.execute_create_multipart_upload_from_s3s(initial_create_req)
.await
.expect("initial conditional create multipart upload should succeed")
.output
@@ -1561,13 +1561,13 @@ mod tests {
.headers
.insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*"));
let create_err = usecase
.execute_create_multipart_upload(second_create_req)
.execute_create_multipart_upload_from_s3s(second_create_req)
.await
.expect_err("create multipart upload with If-None-Match: * on existing key must fail");
assert_eq!(create_err.code(), &crate::app::storage_api::s3::S3ErrorCode::PreconditionFailed);
let second_upload_id = usecase
.execute_create_multipart_upload(build_request(create_input, Method::POST))
.execute_create_multipart_upload_from_s3s(build_request(create_input, Method::POST))
.await
.expect("unconditional create multipart upload should succeed")
.output
@@ -1770,7 +1770,7 @@ mod tests {
.headers
.insert(http::header::IF_NONE_MATCH, HeaderValue::from_static("*"));
let dm_upload_id = usecase
.execute_create_multipart_upload(dm_create_req)
.execute_create_multipart_upload_from_s3s(dm_create_req)
.await
.expect("If-None-Match: * create must succeed past a delete marker")
.output
+304 -71
View File
@@ -15,16 +15,15 @@
//! CreateMultipartUpload.
use super::*;
use crate::app::storage_api::s3::dto::{
ChecksumAlgorithm, ChecksumType, CreateMultipartUploadInput, CreateMultipartUploadOutput,
};
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::RequestEnvelope;
use crate::app::object::shared::build_put_like_object_lock_metadata_native;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
fn create_multipart_upload_metadata(
input_metadata: Option<HashMap<String, String>>,
headers: &HeaderMap,
tagging: Option<String>,
storage_class: Option<&crate::app::storage_api::s3::dto::StorageClass>,
storage_class: Option<&str>,
) -> HashMap<String, String> {
let mut metadata = input_metadata.unwrap_or_default();
namespace_reserved_user_metadata(&mut metadata);
@@ -34,7 +33,7 @@ fn create_multipart_upload_metadata(
metadata.insert(AMZ_OBJECT_TAGGING.to_owned(), tags);
}
if let Some(storage_class) = storage_class {
metadata.insert(AMZ_STORAGE_CLASS.to_owned(), storage_class.as_str().to_owned());
metadata.insert(AMZ_STORAGE_CLASS.to_owned(), storage_class.to_owned());
}
metadata
@@ -63,32 +62,37 @@ fn should_advertise_session_compression(multipart_enabled: bool, ciphertext_pass
}
impl DefaultMultipartUsecase {
#[instrument(level = "debug", skip(self, req))]
#[instrument(level = "debug", skip(self, envelope, input, query_version_id))]
#[hotpath::measure(impl_type = "MultipartUsecase")]
pub async fn execute_create_multipart_upload(
pub(crate) async fn execute_create_multipart_upload(
&self,
req: S3Request<CreateMultipartUploadInput>,
) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
validate_new_object_key(&req.input.key, "CreateMultipartUpload")?;
envelope: RequestEnvelope,
input: CreateMultipartUploadFields,
query_version_id: Option<String>,
) -> S3Result<CreateMultipartUploadResult> {
validate_new_object_key(&input.key, "CreateMultipartUpload")?;
let headers = envelope.headers();
let multipart_max_total_object_size = parse_presigned_multipart_max_total_object_size(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
headers,
envelope.uri().query(),
envelope.verified_sigv4().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedPresignedRequest>().is_some(),
headers,
envelope.uri().query(),
envelope.verified_presigned().is_some(),
)?;
let helper =
OperationHelper::new(&req, EventName::ObjectCreatedCreateMultipartUpload, S3Operation::CreateMultipartUpload)
.suppress_event();
let replication_authorized = replication_request_authorized(&req);
let CreateMultipartUploadInput {
let helper = OperationHelper::from_view(
&envelope.audit_view(),
EventName::ObjectCreatedCreateMultipartUpload,
S3Operation::CreateMultipartUpload,
)
.suppress_event();
let replication_authorized = replication_request_authorized(&envelope);
let CreateMultipartUploadFields {
bucket,
key,
tagging,
version_id,
storage_class,
server_side_encryption,
sse_customer_algorithm,
@@ -99,24 +103,21 @@ impl DefaultMultipartUsecase {
object_lock_mode,
object_lock_retain_until_date,
metadata: input_metadata,
..
} = req.input.clone();
} = input;
let server_side_encryption = server_side_encryption
.map(|value| value.as_str().to_owned())
.or(extract_server_side_encryption_from_headers(&req.headers)?);
let server_side_encryption = server_side_encryption.or(extract_server_side_encryption_from_headers(headers)?);
let ssekms_key_id = ssekms_key_id.or_else(|| {
req.headers
headers
.get("x-amz-server-side-encryption-aws-kms-key-id")
.and_then(|value| value.to_str().ok())
.map(ToOwned::to_owned)
});
// Validate storage class if provided
if let Some(ref storage_class) = storage_class
&& !is_valid_storage_class(storage_class.as_str())
if let Some(storage_class) = storage_class.as_deref()
&& !is_valid_storage_class(storage_class)
{
return Err(s3_error!(InvalidStorageClass));
return Err(s3_error!(InvalidStorageClass, "The storage class you specified is not valid."));
}
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &key).await?;
@@ -128,11 +129,11 @@ impl DefaultMultipartUsecase {
validate_archive_content_encoding(
&key,
req.headers.get("content-type").and_then(|value| value.to_str().ok()),
req.headers.get("content-encoding").and_then(|value| value.to_str().ok()),
headers.get("content-type").and_then(|value| value.to_str().ok()),
headers.get("content-encoding").and_then(|value| value.to_str().ok()),
)?;
let mut metadata = create_multipart_upload_metadata(input_metadata, &req.headers, tagging, storage_class.as_ref());
let mut metadata = create_multipart_upload_metadata(input_metadata, headers, tagging, storage_class.as_deref());
if let Some(limit) = multipart_max_total_object_size {
insert_str(&mut metadata, SUFFIX_MAX_TOTAL_OBJECT_SIZE, limit.to_string());
}
@@ -142,9 +143,9 @@ impl DefaultMultipartUsecase {
let has_explicit_object_lock_retention = object_lock_mode.is_some()
|| object_lock_retain_until_date.is_some()
|| has_replication_retention_update(&req.headers, replication_authorized);
|| has_replication_retention_update(headers, replication_authorized);
let object_lock_config_state = load_bucket_object_lock_config_state(&bucket).await?;
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata(
if let Some(object_lock_metadata) = build_put_like_object_lock_metadata_native(
&bucket,
&object_lock_config_state,
object_lock_legal_hold_status,
@@ -160,20 +161,20 @@ impl DefaultMultipartUsecase {
has_explicit_object_lock_retention,
)?;
let (header_sse_customer_algorithm, header_sse_customer_key, header_sse_customer_key_md5) =
extract_ssec_params_from_headers(&req.headers)?;
extract_ssec_params_from_headers(headers)?;
let sse_customer_algorithm = sse_customer_algorithm.or(header_sse_customer_algorithm);
let sse_customer_key = sse_customer_key.or(header_sse_customer_key);
let sse_customer_key_md5 = sse_customer_key_md5.or(header_sse_customer_key_md5);
// The session data key is generated here, so this is where a multipart upload is held
// to the KMS key it names. Parts and completion also authorize their own caller.
let session_principal = SseKmsPrincipal::from_request(&req);
let session_principal = SseKmsPrincipal::from_request(&envelope);
let encryption_request = PrepareEncryptionRequest {
bucket: &bucket,
key: &key,
server_side_encryption,
ssekms_key_id,
ssekms_context: extract_ssekms_context_from_headers(&req.headers)?,
ssekms_context: extract_ssekms_context_from_headers(headers)?,
sse_customer_algorithm: sse_customer_algorithm.clone(),
sse_customer_key,
sse_customer_key_md5: sse_customer_key_md5.clone(),
@@ -184,8 +185,8 @@ impl DefaultMultipartUsecase {
// session DEK is prepared; a session marker tells UploadPart to store
// the ciphertext verbatim instead of recovering encryption material.
let ciphertext_passthrough = replication_authorized
&& get_header(&req.headers, SUFFIX_SOURCE_REPLICATION_REQUEST).as_deref() == Some("true")
&& rustfs_utils::http::ssec_transport_to_stored_metadata(&req.headers).is_some();
&& get_header(headers, SUFFIX_SOURCE_REPLICATION_REQUEST).as_deref() == Some("true")
&& rustfs_utils::http::ssec_transport_to_stored_metadata(headers).is_some();
if ciphertext_passthrough && let Some(metadata_sys) = self.bucket_metadata_sys() {
let check_result = map_quota_check_outcome(
&bucket,
@@ -228,7 +229,7 @@ impl DefaultMultipartUsecase {
if should_advertise_session_compression(
is_multipart_disk_compression_enabled(),
ciphertext_passthrough,
is_disk_compressible(&req.headers, &key),
is_disk_compressible(headers, &key),
) {
rustfs_utils::http::insert_str(
&mut metadata,
@@ -239,10 +240,10 @@ impl DefaultMultipartUsecase {
let mt2 = metadata.clone();
let mut opts: ObjectOptions =
put_opts_with_replication_authorization(&bucket, &key, version_id, &req.headers, metadata, replication_authorized)
put_opts_with_replication_authorization(&bucket, &key, query_version_id, headers, metadata, replication_authorized)
.await
.map_err(ApiError::from)?;
apply_bucket_generation_guard(&req, &bucket, &mut opts)?;
apply_bucket_generation_guard(&envelope, &bucket, &mut opts)?;
let dsc =
must_replicate_object(&bucket, &key, &mt2, "".to_string(), opts.delete_marker_replication_status(), opts.clone())
@@ -261,7 +262,7 @@ impl DefaultMultipartUsecase {
&& super::super::object::shared::object_lock_checks_required_for_state(&object_lock_config_state)
{
let current_opts: ObjectOptions = internal_object_info_lookup_opts(
get_opts(&bucket, &key, opts.version_id.clone(), None, &req.headers)
get_opts(&bucket, &key, opts.version_id.clone(), None, headers)
.await
.map_err(ApiError::from)?,
);
@@ -277,7 +278,7 @@ impl DefaultMultipartUsecase {
}
}
let checksum_type = rustfs_rio::ChecksumType::from_header(&req.headers);
let checksum_type = rustfs_rio::ChecksumType::from_header(headers);
if checksum_type.is(rustfs_rio::ChecksumType::INVALID) {
return Err(s3_error!(InvalidArgument, "Invalid checksum type"));
} else if checksum_type.is_set() && !checksum_type.is(rustfs_rio::ChecksumType::TRAILING) {
@@ -299,21 +300,20 @@ impl DefaultMultipartUsecase {
.await
.map_err(ApiError::from)?;
let output = CreateMultipartUploadOutput {
bucket: Some(bucket),
key: Some(key),
upload_id: Some(upload_id),
server_side_encryption: effective_sse.map(Into::into),
let output = CreateMultipartUploadResult {
bucket,
key,
upload_id,
server_side_encryption: effective_sse,
sse_customer_algorithm,
sse_customer_key_md5,
ssekms_key_id: effective_kms_key_id,
checksum_algorithm: checksum_algo.map(ChecksumAlgorithm::from),
checksum_type: checksum_type.map(ChecksumType::from),
..Default::default()
checksum_algorithm: checksum_algo,
checksum_type,
};
let result = Ok(S3Response::new(output));
let _ = helper.complete(&result);
let result = Ok(output);
let _ = helper.complete_owned(&result, http::StatusCode::OK, &HeaderMap::new());
result
}
}
@@ -322,6 +322,51 @@ impl DefaultMultipartUsecase {
mod tests {
use super::super::test_support::*;
use super::*;
use crate::app::storage_api::s3::{S3ErrorCode, dto::CreateMultipartUploadInput};
#[tokio::test]
async fn create_multipart_baseline_legacy_direct() {
use crate::app::storage_api::s3::dto::Timestamp;
let date = time::Date::from_calendar_date(-1, time::Month::January, 1)
.unwrap()
.midnight()
.assume_utc();
let long_key = "k".repeat(1025);
let invalid_class = ("InvalidStorageClass", "The storage class you specified is not valid.");
for (bucket, key, invalid, date, expected) in [
("ab", "object", true, None, invalid_class),
("a/b", "object", true, None, invalid_class),
("bucket", "", true, None, ("InvalidArgument", "Invalid object key")),
(
"bucket",
"a\0b",
true,
None,
("InvalidArgument", "Object key contains invalid control characters: \"a\\0b\""),
),
("bucket", long_key.as_str(), true, None, invalid_class),
("bucket", "object", false, Some(date), ("InternalError", "Not init")),
("bucket", "object", true, Some(date), invalid_class),
] {
let input = CreateMultipartUploadInput::builder()
.bucket(bucket.to_owned())
.key(key.to_owned())
.storage_class(invalid.then(|| StorageClass::from("unknown-class".to_owned())))
.object_lock_retain_until_date(date.map(Timestamp::from))
.build()
.expect("direct fixture DTO");
let error = make_usecase()
.execute_create_multipart_upload_from_s3s(build_request(input, Method::POST))
.await
.expect_err("direct fixture does not create an upload");
assert_eq!(
(error.code().as_str(), error.message()),
(expected.0, Some(expected.1)),
"{bucket:?}/{key:?}"
);
}
}
#[test]
fn session_compression_is_advertised_only_for_non_passthrough_compressible_uploads() {
@@ -358,18 +403,16 @@ mod tests {
.unwrap();
let req = build_request(input, Method::POST);
let err = make_usecase().execute_create_multipart_upload(req).await.unwrap_err();
let err = make_usecase()
.execute_create_multipart_upload_from_s3s(req)
.await
.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidStorageClass);
}
#[test]
fn create_multipart_upload_metadata_persists_dto_storage_class_without_raw_header() {
let metadata = create_multipart_upload_metadata(
None,
&HeaderMap::new(),
None,
Some(&StorageClass::from_static("REDUCED_REDUNDANCY")),
);
let metadata = create_multipart_upload_metadata(None, &HeaderMap::new(), None, Some("REDUCED_REDUNDANCY"));
assert_eq!(metadata.get(AMZ_STORAGE_CLASS), Some(&"REDUCED_REDUNDANCY".to_string()));
}
@@ -388,7 +431,7 @@ mod tests {
Some(input_metadata),
&headers,
Some("project=rustfs".to_string()),
Some(&StorageClass::from_static("REDUCED_REDUNDANCY")),
Some("REDUCED_REDUNDANCY"),
);
assert_eq!(metadata.get("content-type"), Some(&"application/octet-stream".to_string()));
@@ -401,10 +444,10 @@ mod tests {
#[tokio::test]
#[serial_test::serial]
async fn create_multipart_rejects_ciphertext_replication_before_parts_are_staged() {
let (_store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("ciphertext-multipart-quota", 4096).await;
let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("ciphertext-multipart-quota", 4096).await;
let usecase = DefaultMultipartUsecase::from_global();
let input = CreateMultipartUploadInput::builder()
.bucket(bucket)
.bucket(bucket.clone())
.key("object".to_string())
.build()
.expect("create multipart request should build");
@@ -418,10 +461,200 @@ mod tests {
..Default::default()
});
let err = usecase
.execute_create_multipart_upload(request)
.await
.expect_err("quota-enabled ciphertext multipart replication should fail before upload creation");
let result = usecase.execute_create_multipart_upload_from_s3s(request).await;
assert!(
store
.list_multipart_uploads(&bucket, "", None, None, None, 100)
.await
.expect("list durable uploads")
.uploads
.is_empty()
);
let err = result.expect_err("quota-enabled ciphertext multipart replication must be refused");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
assert_eq!(
err.message(),
Some("SSE-C ciphertext replication is unavailable for quota-enabled buckets")
);
}
#[test]
#[serial_test::serial]
fn create_multipart_checks_kms_permission_before_creating_a_session() {
crate::app::gating_test_env::run_large_stack_test("create-multipart-kms-policy", || async {
use crate::app::storage_api::s3::dto::ServerSideEncryption;
use crate::app::storage_api::test::{
ReqInfo,
sse::{RecordingKmsKeyAuthorizer, TestAuthorizationHooks},
};
use rustfs_policy::policy::action::KmsAction;
let (store, bucket) = crate::app::gating_test_env::durable_quota_test_bucket("create-multipart-kms", 16_384).await;
let context = crate::app::runtime_sources::current_app_context().expect("test app context");
let usecase = DefaultMultipartUsecase::with_context(Some(context.clone()));
let key_id = "multipart-test-key";
let kms = context.kms().handle();
kms.reconfigure(rustfs_kms::config::KmsConfig::static_kms(
key_id.to_owned(),
base64_simd::STANDARD.encode_to_string([1_u8; 32]),
))
.await
.expect("configure test KMS");
for allowed in [false, true] {
let input = CreateMultipartUploadInput {
bucket: bucket.clone(),
key: "object".to_owned(),
server_side_encryption: Some(ServerSideEncryption::from_static("aws:kms")),
ssekms_key_id: Some(key_id.to_owned()),
..Default::default()
};
let authorizer = RecordingKmsKeyAuthorizer::new(allowed);
let mut req = build_request(input, Method::POST);
req.extensions.insert(ReqInfo {
cred: Some(rustfs_credentials::Credentials {
access_key: "test-kms-writer".to_owned(),
..Default::default()
}),
..Default::default()
});
req.extensions.insert(TestAuthorizationHooks {
enforced: true,
authorizer: authorizer.clone(),
});
let result = usecase.execute_create_multipart_upload_from_s3s(req).await;
let uploads = store
.list_multipart_uploads(&bucket, "", None, None, None, 100)
.await
.expect("list uploads after KMS decision")
.uploads;
assert_eq!(uploads.len(), usize::from(allowed));
assert_eq!(authorizer.calls(), [(KmsAction::GenerateDataKeyAction, key_id.to_owned())]);
if allowed {
let output = result.expect("KMS authorizes the selected key").output;
assert_eq!(output.upload_id.as_deref(), Some(uploads[0].upload_id.as_str()));
} else {
let error = result.expect_err("KMS denied the selected key");
assert_eq!(error.code(), &S3ErrorCode::AccessDenied);
}
}
kms.stop().await.expect("stop test KMS");
});
}
#[test]
fn create_multipart_preserves_legacy_date_precision() {
use time::{Date, Month, Time};
let date = Date::from_calendar_date(-1, Month::January, 1)
.unwrap()
.with_time(Time::from_hms_nano(0, 0, 0, 123_456_789).unwrap())
.assume_utc();
let input = CreateMultipartUploadInput {
bucket: "sthree-multipart".to_owned(),
key: "object".to_owned(),
object_lock_retain_until_date: Some(date.into()),
..Default::default()
};
let fields = CreateMultipartUploadFields::from_legacy(input);
assert_eq!(fields.object_lock_retain_until_date, Some(date));
}
#[test]
#[serial_test::serial]
fn create_multipart_generation_fences_the_durable_upload() {
crate::app::gating_test_env::run_large_stack_test("create-multipart-generation", || async {
use crate::app::storage_api::multipart_usecase::access::{AuthorizationRequest, load_bucket_generation_from_store};
use crate::app::storage_api::test::contract::bucket::{
BucketOperations as _, DeleteBucketOptions, MakeBucketOptions,
};
use crate::runtime_sources::ServerContextSlot;
let context = crate::app::gating_test_env::shared_gating_ambient().await;
let store = context.object_store();
let usecase = DefaultMultipartUsecase::with_context(Some(context.clone()));
let slot = ServerContextSlot::new();
assert!(slot.install(context));
let bucket = format!("create-multipart-{}", Uuid::new_v4().simple());
let other = format!("create-multipart-{}", Uuid::new_v4().simple());
let request = |bucket: &str| {
let input = CreateMultipartUploadInput {
bucket: bucket.to_owned(),
key: "object".to_owned(),
..Default::default()
};
let mut request = build_request(input, Method::POST);
request.uri = format!("/{bucket}/object?uploads").parse().expect("fixture URI");
request.extensions.insert(slot.clone());
request
};
for name in [&bucket, &other] {
store
.make_bucket(name, &MakeBucketOptions::default())
.await
.expect("create bucket");
}
let mut stale = request(&bucket);
let guard = load_bucket_generation_from_store(&store, &stale, &bucket)
.await
.expect("capture original incarnation");
stale.install_bucket_generation(guard);
let original = store.bucket_incarnation_id(&bucket).await.expect("old bucket generation");
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete original bucket");
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
let current = store.bucket_incarnation_id(&bucket).await.expect("new bucket generation");
assert_ne!(original, current);
let missing = request(&bucket);
let mut wrong = request(&bucket);
let guard = load_bucket_generation_from_store(&store, &wrong, &other)
.await
.expect("capture the other bucket");
wrong.install_bucket_generation(guard);
for (request, code) in [
(stale, S3ErrorCode::NoSuchBucket),
(missing, S3ErrorCode::InternalError),
(wrong, S3ErrorCode::InternalError),
] {
let result = usecase.execute_create_multipart_upload_from_s3s(request).await;
assert!(
store
.list_multipart_uploads(&bucket, "", None, None, None, 100)
.await
.expect("list uploads in the recreated bucket")
.uploads
.is_empty()
);
let error = result.expect_err("creation requires the target incarnation");
assert_eq!(error.code(), &code);
}
let probe = crate::app::storage_api::multipart_usecase::helper::MutationCompletionProbe::install(&bucket);
let mut allowed = request(&bucket);
let guard = load_bucket_generation_from_store(&store, &allowed, &bucket)
.await
.expect("capture current incarnation");
allowed.install_bucket_generation(guard);
let response = usecase
.execute_create_multipart_upload_from_s3s(allowed)
.await
.expect("a current guard allows creation");
let uploads = store
.list_multipart_uploads(&bucket, "", None, None, None, 100)
.await
.expect("list durable uploads")
.uploads;
assert_eq!(uploads.len(), 1);
assert_eq!(Some(uploads[0].upload_id.as_str()), response.output.upload_id.as_deref());
assert_eq!(probe.audits.lock().unwrap().len(), 1);
assert!(probe.events.lock().unwrap().is_empty());
});
}
}
@@ -0,0 +1,48 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Values consumed and produced by multipart initiation.
use std::collections::HashMap;
use time::OffsetDateTime;
/// Keeps the decoded values in their original form until the use case validates them.
// The customer key must never appear in diagnostics.
pub(crate) struct CreateMultipartUploadFields {
pub(crate) bucket: String,
pub(crate) key: String,
pub(crate) metadata: Option<HashMap<String, String>>,
pub(crate) tagging: Option<String>,
pub(crate) storage_class: Option<String>,
pub(crate) server_side_encryption: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) object_lock_legal_hold_status: Option<String>,
pub(crate) object_lock_mode: Option<String>,
pub(crate) object_lock_retain_until_date: Option<OffsetDateTime>,
}
pub(crate) struct CreateMultipartUploadResult {
pub(crate) bucket: String,
pub(crate) key: String,
pub(crate) upload_id: String,
pub(crate) server_side_encryption: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) checksum_algorithm: Option<String>,
pub(crate) checksum_type: Option<String>,
}
+187 -154
View File
@@ -15,17 +15,43 @@
//! UploadPart.
use super::*;
use crate::app::storage_api::s3::dto::{
ChecksumAlgorithm, ServerSideEncryption, StreamingBlob, UploadPartInput, UploadPartOutput,
};
use crate::app::storage_api::s3::stream::ByteStream;
use crate::app::storage_api::s3::{S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use crate::app::RequestEnvelope;
use crate::app::object::BodyChunks;
use crate::app::object::request_body::{DemandReader, UploadChunks, guard_native_upload_progress};
use crate::app::storage_api::multipart_usecase::access::AuthorizationRequest;
use crate::app::storage_api::multipart_usecase::contract::multipart::MultipartInfo;
use http_body::Body as _;
use rustfs_gateway_stream::Body;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{RecordedUpload, resolve_upload_async};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
fn resolve_upload_part_size(content_length: Option<i64>, body: Option<&StreamingBlob>) -> rustfs_s3_types::S3Result<Option<i64>> {
#[cfg(any(feature = "gateway-stack", test))]
struct PartUpload(MultipartInfo);
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for PartUpload {
fn bucket(&self) -> &str {
&self.0.bucket
}
fn key(&self) -> &str {
&self.0.object
}
}
async fn lookup_part_upload(store: &ECStore, bucket: &str, key: &str, id: &str, opts: &ObjectOptions) -> S3Result<MultipartInfo> {
let result = store.get_multipart_info(bucket, key, id, opts).await;
#[cfg(test)]
let result = tests::observe_lookup(result);
result.map_err(|error| multipart_upload_api_error(error).into())
}
fn resolve_upload_part_size(content_length: Option<i64>, body: Option<&Body>) -> rustfs_s3_types::S3Result<Option<i64>> {
if let Some(length) = content_length {
return Ok(Some(length));
}
body.and_then(|body| body.remaining_length().exact())
body.and_then(|body| body.size_hint().exact())
.map(i64::try_from)
.transpose()
.map_err(|_| rustfs_s3_types::s3_error!(UnexpectedContent))
@@ -59,28 +85,29 @@ fn passthrough_part_actual_size(headers: &HeaderMap) -> Option<i64> {
// Hoisted values are signed query parameters, not HTTP headers. Normalize only
// after access control has verified the presigned request.
fn normalize_presigned_part_checksums(req: &mut S3Request<UploadPartInput>) -> S3Result<()> {
if req.extensions.get::<VerifiedPresignedRequest>().is_none() {
fn normalize_presigned_part_checksums(req: &mut RequestEnvelope, input: &mut UploadPartWriteInput) -> S3Result<()> {
if req.verified_presigned().is_none() {
return Ok(());
}
let Some(query) = req.uri.query() else {
let uri = req.uri().clone();
let Some(query) = uri.query() else {
return Ok(());
};
let mut seen = std::collections::HashSet::new();
for (name, value) in url::form_urlencoded::parse(query.as_bytes()) {
let field = match name.as_ref() {
"x-amz-checksum-crc32" => Some(&mut req.input.checksum_crc32),
"x-amz-checksum-crc32c" => Some(&mut req.input.checksum_crc32c),
"x-amz-checksum-crc64nvme" => Some(&mut req.input.checksum_crc64nvme),
"x-amz-checksum-sha1" => Some(&mut req.input.checksum_sha1),
"x-amz-checksum-sha256" => Some(&mut req.input.checksum_sha256),
"x-amz-checksum-crc32" => Some(&mut input.checksum_crc32),
"x-amz-checksum-crc32c" => Some(&mut input.checksum_crc32c),
"x-amz-checksum-crc64nvme" => Some(&mut input.checksum_crc64nvme),
"x-amz-checksum-sha1" => Some(&mut input.checksum_sha1),
"x-amz-checksum-sha256" => Some(&mut input.checksum_sha256),
"x-amz-sdk-checksum-algorithm" => None,
_ => continue,
};
if !seen.insert(name.clone()) {
return Err(S3Error::with_message(S3ErrorCode::InvalidRequest, "Duplicate checksum query parameter"));
}
if let Some(header) = req.headers.get(name.as_ref())
if let Some(header) = req.headers().get(name.as_ref())
&& header.as_bytes() != value.as_bytes()
{
return Err(S3Error::with_message(
@@ -90,7 +117,7 @@ fn normalize_presigned_part_checksums(req: &mut S3Request<UploadPartInput>) -> S
}
let header = http::HeaderValue::from_str(&value)
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "Invalid checksum query parameter"))?;
req.headers.insert(
req.headers_mut().insert(
http::header::HeaderName::from_bytes(name.as_bytes())
.map_err(|_| S3Error::with_message(S3ErrorCode::InvalidArgument, "Invalid checksum query parameter"))?,
header,
@@ -98,59 +125,62 @@ fn normalize_presigned_part_checksums(req: &mut S3Request<UploadPartInput>) -> S
if let Some(field) = field {
*field = Some(value.into_owned());
} else {
req.input.checksum_algorithm = Some(ChecksumAlgorithm::from(value.into_owned()));
input.checksum_algorithm = Some(value.into_owned());
}
}
Ok(())
}
impl DefaultMultipartUsecase {
#[instrument(level = "debug", skip(self, req))]
#[instrument(name = "execute_upload_part", level = "debug", skip(self, req, input))]
#[hotpath::measure(impl_type = "MultipartUsecase")]
pub async fn execute_upload_part(&self, mut req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
pub(crate) async fn execute_upload_part_owned(
&self,
mut req: RequestEnvelope,
mut input: UploadPartWriteInput,
) -> S3Result<UploadPartWriteOutput> {
reject_presigned_multipart_max_total_object_size_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedSigV4Request>().is_some(),
req.headers(),
req.uri().query(),
req.verified_sigv4().is_some(),
)?;
reject_presigned_put_max_content_length_for_other_operation(
&req.headers,
req.uri.query(),
req.extensions.get::<VerifiedPresignedRequest>().is_some(),
req.headers(),
req.uri().query(),
req.verified_presigned().is_some(),
)?;
normalize_presigned_part_checksums(&mut req)?;
normalize_presigned_part_checksums(&mut req, &mut input)?;
let (bucket, key) = input.target.resource();
let mut opts = ObjectOptions::default();
apply_bucket_generation_guard(&req, &req.input.bucket, &mut opts)?;
apply_bucket_generation_guard(&req, bucket, &mut opts)?;
let part_principal = SseKmsPrincipal::from_request(&req);
let input = req.input;
let UploadPartInput {
let UploadPartWriteInput {
body,
bucket,
key,
upload_id,
frame_trailers,
trailing_headers,
part_number,
content_length,
sse_customer_algorithm,
sse_customer_key,
sse_customer_key_md5,
// content_md5,
..
} = input;
let part_id = parse_upload_part_number(part_number)?;
reject_veeam_sosapi_mutation(|| self.object_store(), &bucket, &key).await?;
validate_table_catalog_object_mutation(&bucket, &key).await?;
reject_veeam_sosapi_mutation(|| self.object_store(), bucket, key).await?;
validate_table_catalog_object_mutation(bucket, key).await?;
// cuObject S3-over-RDMA UploadPart: an empty HTTP body plus
// `x-amz-rdma-token` means the part bytes are read from the client's
// registered buffer (see `object::cuobject::rdma_put_body`).
#[cfg(all(any(feature = "cuobject", feature = "nixl"), target_os = "linux"))]
let (body, content_length, rdma_put) = match crate::app::object::cuobject::rdma_token_for_request(&req.headers) {
let (body, content_length, rdma_put) = match crate::app::object::cuobject::rdma_token_for_request(req.headers()) {
Some(token) if content_length.unwrap_or(0) == 0 => {
let (body, total) = crate::app::object::cuobject::rdma_put_body(&bucket, &key, &token, true)?;
let (body, total) =
crate::app::object::cuobject::rdma_put_body(bucket, key, &token, true).map_err(S3Error::from)?;
let total = i64::try_from(total).map_err(|_| S3Error::new(S3ErrorCode::EntityTooLarge))?;
(Some(body), Some(total), Some(total))
(Some(crate::app::object::legacy_put_body(body)), Some(total), Some(total))
}
_ => (body, content_length, None),
};
@@ -165,10 +195,29 @@ impl DefaultMultipartUsecase {
return Err(S3Error::with_message(S3ErrorCode::InternalError, "Not init".to_string()));
};
let multipart_info_started = rustfs_io_metrics::put_stage_timer();
let fi = store
.get_multipart_info(&bucket, &key, &upload_id, &opts)
.await
.map_err(multipart_upload_api_error)?;
let (upload_id, fi) = match &input.target {
UploadPartTarget::Legacy { upload_id, .. } => {
let info = lookup_part_upload(&store, bucket, key, upload_id, &opts).await?;
(upload_id.clone(), info)
}
#[cfg(any(feature = "gateway-stack", test))]
UploadPartTarget::Gateway {
bucket: bucket_name,
key: object_key,
upload_id,
} => {
let lookup_store = &store;
let lookup_opts = &opts;
let (id, upload) = resolve_upload_async(upload_id, bucket_name, object_key, |id| async move {
lookup_part_upload(lookup_store, bucket, key, id, lookup_opts)
.await
.map(|info| Some(PartUpload(info)))
})
.await?
.map_err(|rejection| S3Error::with_message(S3ErrorCode::NoSuchUpload, rejection.reason()))?;
(id.id().to_owned(), upload.0)
}
};
rustfs_io_metrics::record_put_object_stage_duration_from("multipart_get_info", multipart_info_started);
authorize_sse_kms_multipart_write(part_principal.as_ref(), &fi.user_defined).await?;
let max_total_object_size = multipart_max_total_object_size(&fi.user_defined)?;
@@ -197,26 +246,16 @@ impl DefaultMultipartUsecase {
));
}
};
let request_id = req
.extensions
.get::<super::super::storage_api::multipart_usecase::request_context::RequestContext>()
.map(|ctx| ctx.request_id.as_str())
.unwrap_or_default();
let (chunks, _) = BodyChunks::with_trailers(body_stream, frame_trailers);
let body_stream: UploadChunks = Box::new(chunks);
let request_id = req.request_context().map(|ctx| ctx.request_id.as_str()).unwrap_or_default();
let timeout = upload_part_body_read_timeout(put_object_body_read_timeout(), max_total_object_size.is_some());
let (body_stream, progress_control) =
if let Some(control) = req.extensions.get::<super::super::object::request_body::BodyReadControl>() {
control.activate_upload(timeout, &bucket, &key, request_id, size.max(0) as u64);
(body_stream, control.clone())
} else {
super::super::object::request_body::guard_upload_progress(
body_stream,
timeout,
&bucket,
&key,
request_id,
size.max(0) as u64,
)
};
let (body_stream, progress_control) = if let Some(control) = req.body_read_control() {
control.activate_upload(timeout, bucket, key, request_id, size.max(0) as u64);
(body_stream, control.clone())
} else {
guard_native_upload_progress(body_stream, timeout, bucket, key, request_id, size.max(0) as u64)
};
let ingress_stage_start = rustfs_io_metrics::put_stage_metrics_enabled().then(std::time::Instant::now);
@@ -238,7 +277,7 @@ impl DefaultMultipartUsecase {
&& rustfs_utils::http::contains_key_str(&fi.user_defined, rustfs_utils::http::SUFFIX_COMPRESSION);
let actual_size = if preserve_ciphertext {
passthrough_part_actual_size(&req.headers).unwrap_or(size)
passthrough_part_actual_size(req.headers()).unwrap_or(size)
} else {
size
};
@@ -252,7 +291,7 @@ impl DefaultMultipartUsecase {
None
};
let mut sha256hex = get_content_sha256_with_query(&req.headers, req.uri.query());
let mut sha256hex = get_content_sha256_with_query(req.headers(), req.uri().query());
let mut write_plan = WritePlan::new();
let etag_algorithm = multipart_etag_algorithm(&fi.user_defined);
@@ -269,7 +308,7 @@ impl DefaultMultipartUsecase {
)
.map_err(ApiError::from)?;
if let Err(err) = hrd.add_checksum(&req.headers, trailer_source(req.trailing_headers.clone()), false) {
if let Err(err) = hrd.add_checksum(req.headers(), trailing_headers.clone(), false) {
return Err(ApiError::from(err).into());
}
@@ -281,7 +320,7 @@ impl DefaultMultipartUsecase {
.map_err(ApiError::from)?
};
if let Err(err) = reader.add_checksum(&req.headers, trailer_source(req.trailing_headers.clone()), size < 0) {
if let Err(err) = reader.add_checksum(req.headers(), trailing_headers.clone(), size < 0) {
return Err(ApiError::from(err).into());
}
opts.want_checksum = reader.checksum();
@@ -297,7 +336,7 @@ impl DefaultMultipartUsecase {
} else {
let sse = fi.user_defined.get("x-amz-server-side-encryption").cloned();
let key_id = match sse.as_ref() {
Some(sse) if sse.as_str() == ServerSideEncryption::AWS_KMS => fi
Some(sse) if sse.as_str() == "aws:kms" => fi
.user_defined
.get("x-amz-server-side-encryption-aws-kms-key-id")
.map(|s| s.to_string()),
@@ -307,8 +346,8 @@ impl DefaultMultipartUsecase {
};
if !preserve_ciphertext {
EncryptionRequest {
bucket: &bucket,
key: &key,
bucket,
key,
server_side_encryption: server_side_encryption.clone(),
ssekms_key_id: ssekms_key_id.clone(),
ssekms_context: None,
@@ -322,8 +361,8 @@ impl DefaultMultipartUsecase {
}
let (requested_sse, requested_kms_key_id) = if has_ssec {
let ssec_material = sse_decryption(DecryptionRequest {
bucket: &bucket,
key: &key,
bucket,
key,
metadata: &fi.user_defined,
sse_customer_key: sse_customer_key.as_ref(),
sse_customer_key_md5: sse_customer_key_md5.as_ref(),
@@ -343,8 +382,8 @@ impl DefaultMultipartUsecase {
// Reuse the stored envelope after authorizing this caller against its
// pinned destination key above.
let managed_material = sse_decryption(DecryptionRequest {
bucket: &bucket,
key: &key,
bucket,
key,
metadata: &fi.user_defined,
sse_customer_key: None,
sse_customer_key_md5: None,
@@ -369,8 +408,7 @@ impl DefaultMultipartUsecase {
{
use rustfs_rio::HashReaderMut;
let inner = reader.take_inner();
reader.inner =
rustfs_rio::boxed_reader(super::super::object::request_body::DemandReader::new(inner, progress_control));
reader.inner = rustfs_rio::boxed_reader(DemandReader::new(inner, progress_control));
}
let mut reader = PutObjReader::new(reader);
@@ -385,7 +423,7 @@ impl DefaultMultipartUsecase {
let _upload_part_admission = upload_part_admission;
let store_write_started = rustfs_io_metrics::put_stage_timer();
let result = store
.put_object_part(&bucket, &key, &upload_id, part_id, &mut reader, &opts)
.put_object_part(bucket, key, &upload_id, part_id, &mut reader, &opts)
.await;
rustfs_io_metrics::record_put_object_stage_duration_from("multipart_store_write", store_write_started);
drop(_upload_part_admission);
@@ -397,30 +435,21 @@ impl DefaultMultipartUsecase {
let mut checksum_sha256 = input.checksum_sha256;
let mut checksum_crc64nvme = input.checksum_crc64nvme;
if let Some(alg) = &input.checksum_algorithm
&& let Some(Some(checksum_str)) = req.trailing_headers.as_ref().map(|trailer| {
let key = match alg.as_str() {
ChecksumAlgorithm::CRC32 => rustfs_rio::ChecksumType::CRC32.key(),
ChecksumAlgorithm::CRC32C => rustfs_rio::ChecksumType::CRC32C.key(),
ChecksumAlgorithm::SHA1 => rustfs_rio::ChecksumType::SHA1.key(),
ChecksumAlgorithm::SHA256 => rustfs_rio::ChecksumType::SHA256.key(),
ChecksumAlgorithm::CRC64NVME => rustfs_rio::ChecksumType::CRC64_NVME.key(),
_ => return None,
};
trailer.read(|headers| {
headers
.get(key.unwrap_or_default())
.and_then(|value| value.to_str().ok().map(|s| s.to_string()))
})
})
{
match alg.as_str() {
ChecksumAlgorithm::CRC32 => checksum_crc32 = checksum_str,
ChecksumAlgorithm::CRC32C => checksum_crc32c = checksum_str,
ChecksumAlgorithm::SHA1 => checksum_sha1 = checksum_str,
ChecksumAlgorithm::SHA256 => checksum_sha256 = checksum_str,
ChecksumAlgorithm::CRC64NVME => checksum_crc64nvme = checksum_str,
_ => (),
if let (Some(algorithm), Some(trailers)) = (input.checksum_algorithm.as_deref(), trailing_headers) {
let echo = match algorithm {
"CRC32" => Some(("x-amz-checksum-crc32", &mut checksum_crc32)),
"CRC32C" => Some(("x-amz-checksum-crc32c", &mut checksum_crc32c)),
"SHA1" => Some(("x-amz-checksum-sha1", &mut checksum_sha1)),
"SHA256" => Some(("x-amz-checksum-sha256", &mut checksum_sha256)),
"CRC64NVME" => Some(("x-amz-checksum-crc64nvme", &mut checksum_crc64nvme)),
_ => None,
};
if let Some((name, field)) = echo {
match trailers.lookup(name) {
rustfs_rio::TrailerValue::Present(value) => *field = Some(value),
rustfs_rio::TrailerValue::Missing => *field = None,
rustfs_rio::TrailerValue::Pending => {}
}
}
}
@@ -441,8 +470,10 @@ impl DefaultMultipartUsecase {
.push((crate::app::object::cuobject::RDMA_BYTES_TRANSFERRED_HEADER, moved.to_string()));
}
let output = UploadPartOutput {
server_side_encryption: requested_sse.map(ServerSideEncryption::from),
let mut headers = HeaderMap::new();
crate::app::object_usecase::inject_additional_checksum_headers(&mut headers, &upload_part_extra_checksum_headers);
Ok(UploadPartWriteOutput {
server_side_encryption: requested_sse,
ssekms_key_id: requested_kms_key_id,
sse_customer_algorithm,
sse_customer_key_md5,
@@ -451,16 +482,9 @@ impl DefaultMultipartUsecase {
checksum_sha1,
checksum_sha256,
checksum_crc64nvme,
e_tag: info.etag.map(|etag| to_s3s_etag(&etag)),
..Default::default()
};
let mut response = S3Response::new(output);
crate::app::object_usecase::inject_additional_checksum_headers(
&mut response.headers,
&upload_part_extra_checksum_headers,
);
Ok(response)
e_tag: info.etag,
headers,
})
}
}
@@ -468,12 +492,15 @@ impl DefaultMultipartUsecase {
mod tests {
use super::super::test_support::*;
use super::*;
use crate::app::object::legacy_put_body;
use crate::app::storage_api::s3::S3ErrorCode as LegacyErrorCode;
use crate::app::storage_api::s3::dto::{StreamingBlob, UploadPartInput};
fn presigned_checksum_request(query: &str) -> S3Request<UploadPartInput> {
fn presigned_checksum_request(query: &str) -> (RequestEnvelope, UploadPartWriteInput) {
let mut req = build_request(UploadPartInput::default(), Method::PUT);
req.uri = format!("/bucket/object?{query}").parse().unwrap();
req.extensions.insert(VerifiedPresignedRequest);
req
(RequestEnvelope::from_s3s(&req), UploadPartWriteInput::from_legacy(req.input, None))
}
#[tokio::test]
@@ -483,15 +510,12 @@ mod tests {
let query =
"x-amz-checksum-sha256=ungWv48Bz%2BpBQUDeXa4iI7ADYaOWF3qctBD%2FYfIAFa0%3D&x-amz-sdk-checksum-algorithm=SHA256";
for payload in [b"abc", b"abd"] {
let mut req = presigned_checksum_request(query);
normalize_presigned_part_checksums(&mut req).unwrap();
assert_eq!(req.input.checksum_sha256.as_deref(), Some(checksum));
assert_eq!(
req.input.checksum_algorithm.as_ref().map(|algorithm| algorithm.as_str()),
Some(ChecksumAlgorithm::SHA256)
);
let (mut req, mut input) = presigned_checksum_request(query);
normalize_presigned_part_checksums(&mut req, &mut input).unwrap();
assert_eq!(input.checksum_sha256.as_deref(), Some(checksum));
assert_eq!(input.checksum_algorithm.as_deref(), Some("SHA256"));
let mut reader = HashReader::from_stream(Cursor::new(payload), 3, 3, None, None, false).unwrap();
reader.add_checksum(&req.headers, None, false).unwrap();
reader.add_checksum(req.headers(), None, false).unwrap();
assert_eq!(reader.content_crc_type(), Some(rustfs_rio::ChecksumType::SHA256));
let mut bytes = Vec::new();
let result = reader.read_to_end(&mut bytes).await;
@@ -508,11 +532,12 @@ mod tests {
#[tokio::test]
async fn normalize_presigned_part_checksums_accepts_sdk_crc32_default() {
let mut req = presigned_checksum_request("x-amz-checksum-crc32=y%2FQ5Jg%3D%3D&x-amz-sdk-checksum-algorithm=CRC32");
normalize_presigned_part_checksums(&mut req).unwrap();
assert_eq!(req.input.checksum_crc32.as_deref(), Some("y/Q5Jg=="));
let (mut req, mut input) =
presigned_checksum_request("x-amz-checksum-crc32=y%2FQ5Jg%3D%3D&x-amz-sdk-checksum-algorithm=CRC32");
normalize_presigned_part_checksums(&mut req, &mut input).unwrap();
assert_eq!(input.checksum_crc32.as_deref(), Some("y/Q5Jg=="));
let mut reader = HashReader::from_stream(Cursor::new(b"123456789"), 9, 9, None, None, false).unwrap();
reader.add_checksum(&req.headers, None, false).unwrap();
reader.add_checksum(req.headers(), None, false).unwrap();
let mut bytes = Vec::new();
reader.read_to_end(&mut bytes).await.unwrap();
assert_eq!(bytes, b"123456789");
@@ -525,44 +550,47 @@ mod tests {
"x-amz-checksum-sha256=one&x-amz-checksum-sha256=two",
"x-amz-sdk-checksum-algorithm=SHA256&x-amz-sdk-checksum-algorithm=CRC32",
] {
let mut req = presigned_checksum_request(query);
let (mut req, mut input) = presigned_checksum_request(query);
assert_eq!(
*normalize_presigned_part_checksums(&mut req).unwrap_err().code(),
*normalize_presigned_part_checksums(&mut req, &mut input).unwrap_err().code(),
S3ErrorCode::InvalidRequest
);
}
let mut req = presigned_checksum_request("x-amz-checksum-sha256=one");
req.headers.insert("x-amz-checksum-sha256", HeaderValue::from_static("two"));
let (mut req, mut input) = presigned_checksum_request("x-amz-checksum-sha256=one");
req.headers_mut()
.insert("x-amz-checksum-sha256", HeaderValue::from_static("two"));
assert_eq!(
*normalize_presigned_part_checksums(&mut req).unwrap_err().code(),
*normalize_presigned_part_checksums(&mut req, &mut input).unwrap_err().code(),
S3ErrorCode::InvalidRequest
);
}
#[test]
fn normalize_presigned_part_checksums_preserves_headers_and_rejects_invalid_values() {
let mut req = presigned_checksum_request("partNumber=1&uploadId=test");
req.input.checksum_sha256 = Some("existing".to_owned());
req.headers
let (mut req, mut input) = presigned_checksum_request("partNumber=1&uploadId=test");
input.checksum_sha256 = Some("existing".to_owned());
req.headers_mut()
.insert("x-amz-checksum-sha256", HeaderValue::from_static("existing"));
normalize_presigned_part_checksums(&mut req).unwrap();
assert_eq!(req.input.checksum_sha256.as_deref(), Some("existing"));
assert_eq!(req.headers["x-amz-checksum-sha256"], "existing");
normalize_presigned_part_checksums(&mut req, &mut input).unwrap();
assert_eq!(input.checksum_sha256.as_deref(), Some("existing"));
assert_eq!(req.headers()["x-amz-checksum-sha256"], "existing");
let mut req = presigned_checksum_request("x-amz-checksum-sha256=%0D%0A");
let (mut req, mut input) = presigned_checksum_request("x-amz-checksum-sha256=%0D%0A");
assert_eq!(
*normalize_presigned_part_checksums(&mut req).unwrap_err().code(),
*normalize_presigned_part_checksums(&mut req, &mut input).unwrap_err().code(),
S3ErrorCode::InvalidArgument
);
}
#[test]
fn normalize_presigned_part_checksums_requires_verified_presign() {
let mut req = presigned_checksum_request("x-amz-checksum-sha256=one");
req.extensions.remove::<VerifiedPresignedRequest>();
normalize_presigned_part_checksums(&mut req).unwrap();
assert!(req.input.checksum_sha256.is_none());
assert!(req.headers.is_empty());
let mut legacy = build_request(UploadPartInput::default(), Method::PUT);
legacy.uri = "/bucket/object?x-amz-checksum-sha256=one".parse().unwrap();
let mut req = RequestEnvelope::from_s3s(&legacy);
let mut input = UploadPartWriteInput::from_legacy(legacy.input, None);
normalize_presigned_part_checksums(&mut req, &mut input).unwrap();
assert!(input.checksum_sha256.is_none());
assert!(req.headers().is_empty());
}
#[test]
@@ -570,12 +598,15 @@ mod tests {
let result: rustfs_s3_types::S3Result<Option<i64>> = resolve_upload_part_size(Some(5242880), None);
assert_eq!(result.expect("DTO length"), Some(5242880));
assert_eq!(resolve_upload_part_size(None, None).expect("unknown length"), None);
let body = StreamingBlob::from(Bytes::from_static(b"abc"));
let body = Body::from(Bytes::from_static(b"abc"));
assert_eq!(resolve_upload_part_size(None, Some(&body)).expect("exact bytes"), Some(3));
assert_eq!(resolve_upload_part_size(Some(0), Some(&body)).expect("explicit length wins"), Some(0));
let body = StreamingBlob::wrap(futures::stream::iter([Ok::<_, std::io::Error>(Bytes::from_static(b"abc"))]));
assert_eq!(futures::Stream::size_hint(&body), (1, Some(1)), "the stream knows its item count");
assert_eq!(resolve_upload_part_size(None, Some(&body)).expect("unknown byte length"), None);
assert_eq!(
resolve_upload_part_size(None, Some(&legacy_put_body(body))).expect("unknown byte length"),
None
);
}
#[test]
@@ -621,7 +652,7 @@ mod tests {
let req = build_request(input, Method::PUT);
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::IncompleteBody);
assert_eq!(err.code(), &LegacyErrorCode::IncompleteBody);
}
/// issue #7596: a part whose declared length exceeds the 5 GiB
@@ -646,10 +677,10 @@ mod tests {
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
if expect_too_large {
assert_eq!(err.code(), &S3ErrorCode::EntityTooLarge, "declared {declared}");
assert_eq!(err.code(), &LegacyErrorCode::EntityTooLarge, "declared {declared}");
assert_eq!(polls.load(std::sync::atomic::Ordering::SeqCst), 0, "body must not be polled");
} else {
assert_ne!(err.code(), &S3ErrorCode::EntityTooLarge, "declared {declared} must pass admission");
assert_ne!(err.code(), &LegacyErrorCode::EntityTooLarge, "declared {declared} must pass admission");
}
}
}
@@ -667,7 +698,7 @@ mod tests {
let req = build_request(input, Method::PUT);
let err = make_usecase().execute_upload_part(req).await.unwrap_err();
assert_eq!(err.code(), &S3ErrorCode::InvalidArgument);
assert_eq!(err.code(), &LegacyErrorCode::InvalidArgument);
assert_eq!(err.message(), Some("partNumber must be between 1 and 10000"));
}
}
@@ -734,9 +765,9 @@ mod tests {
assert_eq!(
error.code(),
&if capped || content_length.is_some() {
S3ErrorCode::UnexpectedContent
LegacyErrorCode::UnexpectedContent
} else {
S3ErrorCode::MissingContentLength
LegacyErrorCode::MissingContentLength
}
);
assert_eq!(polls.load(std::sync::atomic::Ordering::Relaxed), 0);
@@ -796,7 +827,7 @@ mod tests {
));
control.advance_wait_for_test(Duration::from_secs(60));
sender.send(Ok(Frame::data(Bytes::from_static(b"x")))).unwrap();
assert_eq!(owner.await.unwrap().unwrap_err().code(), &S3ErrorCode::RequestTimeout);
assert_eq!(owner.await.unwrap().unwrap_err().code(), &LegacyErrorCode::RequestTimeout);
assert!(matches!(
manager.admit_put_object(65536).await.unwrap(),
ForegroundWriteAdmission::Admitted(_)
@@ -852,9 +883,11 @@ mod tests {
.execute_upload_part(build_request(input, Method::PUT))
.await
.expect_err("full foreground write admission should reject before body ingest");
assert_eq!(err.code(), &S3ErrorCode::SlowDown);
assert_eq!(err.code(), &LegacyErrorCode::SlowDown);
drop(held);
}
mod body_read_tests;
mod claim_tests;
pub(super) use claim_tests::observe_lookup;
}
@@ -0,0 +1,127 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Upload target ownership and refusal order against real multipart state.
use super::*;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_gateway_types::{BucketName, ObjectKey, UploadIdClaim};
use std::cell::Cell;
#[derive(Clone, Copy, Debug)]
enum LookupResult {
Real,
ForeignBucket,
ForeignKey,
Outage,
}
tokio::task_local! {
static LOOKUP: (Cell<usize>, LookupResult);
}
pub(in crate::app::multipart_usecase::upload_part) fn observe_lookup(
result: Result<MultipartInfo, StorageError>,
) -> Result<MultipartInfo, StorageError> {
let Ok(mode) = LOOKUP.try_with(|(calls, mode)| {
calls.set(calls.get() + 1);
*mode
}) else {
return result;
};
let mut info = result?;
match mode {
LookupResult::Real => {}
LookupResult::ForeignBucket => info.bucket = "foreign-bucket".to_owned(),
LookupResult::ForeignKey => info.object = "foreign-key".to_owned(),
LookupResult::Outage => return Err(StorageError::ErasureReadQuorum),
}
Ok(info)
}
#[tokio::test]
#[serial_test::serial]
async fn native_claim_checks_order_ownership_and_one_active_lookup() {
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
let context = crate::app::gating_test_env::shared_gating_ambient().await;
let usecase = DefaultMultipartUsecase::with_context(Some(context));
let bucket = format!("part-claim-{}", Uuid::new_v4().simple());
let key = "encoded + key/";
let opts = ObjectOptions::default();
store.make_bucket(&bucket, &MakeBucketOptions::default()).await.unwrap();
let active = store.new_multipart_upload(&bucket, key, &opts).await.unwrap().upload_id;
for (number, id, length, body, mode, calls, expected) in [
(1, active.as_str(), Some(3), true, LookupResult::Real, 1, None),
(0, "/impossible", Some(3), true, LookupResult::Real, 0, Some("InvalidArgument")),
(-1, "/impossible", Some(3), true, LookupResult::Real, 0, Some("InvalidArgument")),
(10001, "/impossible", Some(3), true, LookupResult::Real, 0, Some("InvalidArgument")),
(2, "/impossible", Some(3), false, LookupResult::Real, 0, Some("IncompleteBody")),
(2, active.as_str(), Some(3), false, LookupResult::Real, 0, Some("IncompleteBody")),
(2, "/impossible", Some(i64::MAX), true, LookupResult::Real, 0, Some("EntityTooLarge")),
(2, "", Some(3), true, LookupResult::Real, 0, Some("NoSuchUpload")),
(2, "/impossible", Some(-1), true, LookupResult::Real, 0, Some("NoSuchUpload")),
(2, "malformed", None, true, LookupResult::Real, 1, Some("NoSuchUpload")),
(2, "malformed", Some(-1), true, LookupResult::Real, 1, Some("NoSuchUpload")),
(2, active.as_str(), Some(-1), true, LookupResult::Real, 1, Some("UnexpectedContent")),
(2, active.as_str(), None, true, LookupResult::Real, 1, Some("MissingContentLength")),
(2, active.as_str(), Some(3), true, LookupResult::ForeignBucket, 1, Some("NoSuchUpload")),
(2, active.as_str(), Some(3), true, LookupResult::ForeignKey, 1, Some("NoSuchUpload")),
(2, active.as_str(), Some(3), true, LookupResult::Outage, 1, Some("ServiceUnavailable")),
(10000, active.as_str(), Some(3), true, LookupResult::Real, 1, None),
] {
let mut input = UploadPartWriteInput::from_legacy(UploadPartInput::default(), None);
input.target = UploadPartTarget::Gateway {
bucket: BucketName::new(bucket.clone()).unwrap(),
key: ObjectKey::new(key).unwrap(),
upload_id: UploadIdClaim::from_wire(id),
};
input.part_number = number;
input.content_length = length;
input.body = body.then(|| {
if length.is_none() {
legacy_put_body(StreamingBlob::wrap(futures::stream::pending::<Result<Bytes, std::io::Error>>()))
} else {
Body::from(Bytes::from_static(b"abc"))
}
});
let envelope = RequestEnvelope::from_s3s(&build_request((), Method::PUT));
LOOKUP
.scope((Cell::new(0), mode), async {
let result = usecase.execute_upload_part_owned(envelope, input).await;
assert_eq!(
result.as_ref().err().map(|err| err.code().as_str()),
expected,
"part {number}, lookup {mode:?}"
);
assert_eq!(LOOKUP.with(|(observed, _)| observed.get()), calls, "part {number}, lookup {mode:?}");
if let Ok(output) = result {
assert_eq!(output.e_tag.as_deref(), Some("900150983cd24fb0d6963f7d28e17f72"));
}
})
.await;
}
let parts = store
.list_object_parts(&bucket, key, &active, None, 20, &opts)
.await
.unwrap()
.parts;
let parts: Vec<_> = parts
.iter()
.map(|part| (part.part_num, part.size, part.etag.as_deref()))
.collect();
let etag = Some("900150983cd24fb0d6963f7d28e17f72");
assert_eq!(parts, [(1, 3, etag), (10000, 3, etag)]);
}
@@ -0,0 +1,79 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Owned UploadPart values projected by each S3 edge.
use crate::app::object::FrameTrailers;
use rustfs_gateway_stream::Body;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, UploadIdClaim};
use rustfs_rio::SharedTrailerSource;
use std::sync::Arc;
pub(crate) enum UploadPartTarget {
Legacy {
bucket: String,
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
upload_id: UploadIdClaim,
},
}
impl UploadPartTarget {
pub(super) fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
}
pub(crate) struct UploadPartWriteInput {
pub(crate) target: UploadPartTarget,
pub(crate) part_number: i32,
pub(crate) content_length: Option<i64>,
pub(crate) body: Option<Body>,
pub(crate) trailing_headers: Option<SharedTrailerSource>,
pub(crate) frame_trailers: Option<Arc<FrameTrailers>>,
pub(crate) content_md5: Option<String>,
pub(crate) checksum_algorithm: Option<String>,
pub(crate) checksum_crc32: Option<String>,
pub(crate) checksum_crc32c: Option<String>,
pub(crate) checksum_crc64nvme: Option<String>,
pub(crate) checksum_sha1: Option<String>,
pub(crate) checksum_sha256: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
}
pub(crate) struct UploadPartWriteOutput {
pub(crate) e_tag: Option<String>,
pub(crate) checksum_crc32: Option<String>,
pub(crate) checksum_crc32c: Option<String>,
pub(crate) checksum_crc64nvme: Option<String>,
pub(crate) checksum_sha1: Option<String>,
pub(crate) checksum_sha256: Option<String>,
pub(crate) server_side_encryption: Option<String>,
pub(crate) ssekms_key_id: Option<String>,
pub(crate) sse_customer_algorithm: Option<String>,
pub(crate) sse_customer_key_md5: Option<String>,
pub(crate) headers: http::HeaderMap,
}
+1
View File
@@ -224,6 +224,7 @@ mod on_demand_migration_put;
mod put;
mod put_input;
pub(crate) use native_body::from_legacy as legacy_put_body;
pub(crate) use native_body::{BodyChunks, FrameTrailers};
pub(crate) use put_input::{PutObjectWriteInput, PutObjectWriteOutput};
pub(crate) mod request_body;
mod restore;
+2 -2
View File
@@ -108,7 +108,7 @@ impl TrailerSource for FrameTrailers {
}
/// Sync is needed only by RIO's reader trait; polling owns this mutex exclusively.
pub(super) struct BodyChunks {
pub(crate) struct BodyChunks {
body: Mutex<Body>,
trailers: Arc<FrameTrailers>,
ended: bool,
@@ -120,7 +120,7 @@ impl BodyChunks {
Self::with_trailers(body, None)
}
pub(super) fn with_trailers(body: Body, trailers: Option<Arc<FrameTrailers>>) -> (Self, SharedTrailerSource) {
pub(crate) fn with_trailers(body: Body, trailers: Option<Arc<FrameTrailers>>) -> (Self, SharedTrailerSource) {
let trailers = trailers.unwrap_or_default();
(
Self {
+8 -7
View File
@@ -17,7 +17,9 @@
//! after its input returned Pending, so the transport cannot infer read demand.
use super::{LOG_COMPONENT_APP, LOG_SUBSYSTEM_OBJECT};
use crate::app::storage_api::request_body::{ByteStream, DynByteStream, RemainingLength, StdError, StreamingBlob};
use crate::app::storage_api::request_body::StdError;
#[cfg(test)]
use crate::app::storage_api::request_body::{ByteStream, DynByteStream, RemainingLength, StreamingBlob};
use crate::error::ClientBodyReadTimeout;
use bytes::Bytes;
use futures::{Stream, StreamExt};
@@ -421,13 +423,13 @@ impl<R: TryGetIndex> TryGetIndex for DemandReader<R> {
/// Establish demand for eager reads and archive sniffing before the final
/// transformed reader exists. Pending demand is subsequently paused by the
/// outer DemandReader when a transform returns buffered output.
pub(super) type UploadChunks = Box<dyn Stream<Item = Result<Bytes, StdError>> + Send + Sync + Unpin>;
pub(crate) type UploadChunks = Box<dyn Stream<Item = Result<Bytes, StdError>> + Send + Sync + Unpin>;
pub(super) fn demand_native_upload(body: UploadChunks, control: BodyReadControl) -> UploadChunks {
Box::new(DemandStream { inner: body, control })
}
pub(super) fn guard_native_upload_progress(
pub(crate) fn guard_native_upload_progress(
body: UploadChunks,
timeout: Duration,
bucket: &str,
@@ -464,10 +466,8 @@ impl<S: Stream<Item = Result<Bytes, StdError>> + Unpin> Stream for DemandStream<
self.inner.size_hint()
}
}
/// Fallback for direct protocol callers without the raw HTTP observation.
/// HTTP uses ObservedBody because a signed chunk may not yield decoded output
/// until its full payload has arrived and its signature has been verified.
/// The final DemandReader pauses pending reads when transforms return output.
/// Supplies the read-progress guard to tests using a legacy body.
#[cfg(test)]
pub(crate) fn guard_upload_progress(
body: StreamingBlob,
timeout: Duration,
@@ -554,6 +554,7 @@ impl<S: Stream<Item = Result<Bytes, StdError>> + Unpin> Stream for UploadProgres
self.inner.size_hint()
}
}
#[cfg(test)]
impl ByteStream for UploadProgressStream<DynByteStream> {
fn remaining_length(&self) -> RemainingLength {
self.inner.remaining_length()
+6 -3
View File
@@ -28,7 +28,9 @@ pub(crate) fn EndpointServerPools(
pub(crate) mod request_body {
pub(crate) use s3s::StdError;
pub(crate) use s3s::dto::StreamingBlob;
pub(crate) use s3s::stream::{ByteStream, DynByteStream, RemainingLength};
pub(crate) use s3s::stream::ByteStream;
#[cfg(test)]
pub(crate) use s3s::stream::{DynByteStream, RemainingLength};
}
#[cfg(feature = "diagnostic-read-version-caller")]
@@ -125,14 +127,15 @@ pub(crate) mod data_usage {
pub(crate) use crate::storage::storage_api::ecstore_data_usage::quota_object_size;
#[cfg(test)]
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_admin_data_usage_from_backend_cached(
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
}
pub(crate) async fn load_bucket_usage_from_backend_cached(
+68 -10
View File
@@ -12,40 +12,81 @@
// See the License for the specific language governing permissions and
// limitations under the License.
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc", not(target_os = "windows")))]
#[cfg(all(
not(target_os = "windows"),
any(feature = "diagnostic-allocator-live", all(feature = "hotpath", feature = "hotpath-alloc"))
))]
use std::alloc::{GlobalAlloc, Layout};
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc", not(target_os = "windows")))]
#[cfg(all(
not(target_os = "windows"),
any(feature = "diagnostic-allocator-live", all(feature = "hotpath", feature = "hotpath-alloc"))
))]
use std::ptr::NonNull;
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc", not(target_os = "windows")))]
#[cfg(all(
not(target_os = "windows"),
any(feature = "diagnostic-allocator-live", all(feature = "hotpath", feature = "hotpath-alloc"))
))]
#[derive(Default)]
struct MiMallocAllocator;
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc", not(target_os = "windows")))]
#[cfg(all(
not(target_os = "windows"),
any(feature = "diagnostic-allocator-live", all(feature = "hotpath", feature = "hotpath-alloc"))
))]
// SAFETY: allocation operations are forwarded to MiMalloc with the
// corresponding GlobalAlloc size and alignment contracts.
#[allow(unsafe_code)]
unsafe impl GlobalAlloc for MiMallocAllocator {
unsafe fn alloc(&self, layout: Layout) -> *mut u8 {
// SAFETY: the caller upholds GlobalAlloc's contract for layout.
unsafe { rustfs_mimalloc::MiMalloc.alloc(layout) }
let ptr = unsafe { rustfs_mimalloc::MiMalloc.alloc(layout) };
#[cfg(all(
feature = "diagnostic-allocator-live",
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
if !ptr.is_null() {
rustfs::memory_observability::record_diagnostic_global_allocator_resize(0, layout.size());
}
ptr
}
unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8 {
// SAFETY: the caller upholds GlobalAlloc's contract for layout.
unsafe { rustfs_mimalloc::MiMalloc.alloc_zeroed(layout) }
let ptr = unsafe { rustfs_mimalloc::MiMalloc.alloc_zeroed(layout) };
#[cfg(all(
feature = "diagnostic-allocator-live",
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
if !ptr.is_null() {
rustfs::memory_observability::record_diagnostic_global_allocator_resize(0, layout.size());
}
ptr
}
unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout) {
// SAFETY: ptr came from this allocator, is non-null by GlobalAlloc's
// dealloc contract, and layout.size() is the original allocation size.
let ptr = unsafe { NonNull::new_unchecked(ptr) };
unsafe { rustfs_mimalloc::MiMalloc::free_csize_nonnull(ptr, layout.size()) }
unsafe { rustfs_mimalloc::MiMalloc::free_csize_nonnull(ptr, layout.size()) };
#[cfg(all(
feature = "diagnostic-allocator-live",
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
rustfs::memory_observability::record_diagnostic_global_allocator_resize(layout.size(), 0);
}
unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8 {
// SAFETY: ptr and layout came from this allocator and are forwarded unchanged.
unsafe { rustfs_mimalloc::MiMalloc.realloc(ptr, layout, new_size) }
let new_ptr = unsafe { rustfs_mimalloc::MiMalloc.realloc(ptr, layout, new_size) };
#[cfg(all(
feature = "diagnostic-allocator-live",
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
if !new_ptr.is_null() {
rustfs::memory_observability::record_diagnostic_global_allocator_resize(layout.size(), new_size);
}
new_ptr
}
}
@@ -57,10 +98,22 @@ static GLOBAL: hotpath::CountingAllocator<MiMallocAllocator> = hotpath::Counting
#[global_allocator]
static GLOBAL: hotpath::CountingAllocator<std::alloc::System> = hotpath::CountingAllocator::with(std::alloc::System);
#[cfg(all(not(all(feature = "hotpath", feature = "hotpath-alloc")), not(target_os = "windows")))]
#[cfg(all(
not(feature = "diagnostic-allocator-live"),
not(all(feature = "hotpath", feature = "hotpath-alloc")),
not(target_os = "windows")
))]
#[global_allocator]
static GLOBAL: rustfs_mimalloc::MiMalloc = rustfs_mimalloc::MiMalloc;
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
#[global_allocator]
static GLOBAL: MiMallocAllocator = MiMallocAllocator;
fn main() -> std::process::ExitCode {
#[cfg(target_os = "linux")]
#[allow(unsafe_code)]
@@ -73,13 +126,18 @@ fn main() -> std::process::ExitCode {
rustfs::startup_entrypoint::run_process()
}
#[cfg(all(test, feature = "hotpath", feature = "hotpath-alloc", not(target_os = "windows")))]
#[cfg(all(
test,
not(target_os = "windows"),
any(feature = "diagnostic-allocator-live", all(feature = "hotpath", feature = "hotpath-alloc"))
))]
mod tests {
#[test]
// SAFETY: This test inspects a live allocation pointer with mimalloc's heap
// ownership API without dereferencing or extending the pointer lifetime.
#[allow(unsafe_code)]
fn hotpath_allocation_workload_uses_mimalloc() {
#[cfg(all(feature = "hotpath", feature = "hotpath-alloc"))]
let _guard = hotpath::MeasurementGuardSync::new("rustfs::tests::hotpath_allocation_workload_uses_mimalloc", false, false);
let mut allocation = Vec::with_capacity(64);
allocation.extend_from_slice(&[7_u8; 64]);
+145
View File
@@ -20,6 +20,12 @@ use serde::Serialize;
#[cfg(any(not(target_os = "windows"), test))]
use serde_json::Value;
use std::path::Path;
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use tokio_util::sync::CancellationToken;
@@ -35,6 +41,38 @@ const CGROUP_V1_MEMORY_STAT_PATH: &str = "/sys/fs/cgroup/memory/memory.stat";
const CGROUP_V1_MEMORY_USAGE_PATH: &str = "/sys/fs/cgroup/memory/memory.usage_in_bytes";
const CGROUP_V1_MEMORY_LIMIT_PATH: &str = "/sys/fs/cgroup/memory/memory.limit_in_bytes";
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
static DIAGNOSTIC_GLOBAL_ALLOCATOR_LIVE_REQUESTED_BYTES: AtomicUsize = AtomicUsize::new(0);
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
fn adjust_diagnostic_global_allocator_live_bytes(counter: &AtomicUsize, old_size: usize, new_size: usize) {
if new_size > old_size {
counter.fetch_add(new_size - old_size, Ordering::Relaxed);
} else if old_size > new_size {
counter.fetch_sub(old_size - new_size, Ordering::Relaxed);
}
}
/// Update the diagnostic-only live requested-byte counter for Rust allocations
/// routed through the binary's `GlobalAlloc` wrapper.
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
#[doc(hidden)]
pub fn record_diagnostic_global_allocator_resize(old_size: usize, new_size: usize) {
adjust_diagnostic_global_allocator_live_bytes(&DIAGNOSTIC_GLOBAL_ALLOCATOR_LIVE_REQUESTED_BYTES, old_size, new_size);
}
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize)]
#[serde(rename_all = "snake_case")]
pub enum MemoryObservabilityServiceState {
@@ -457,7 +495,46 @@ fn allocator_snapshot_or_unavailable(snapshot: Option<AllocatorMemorySnapshot>)
})
}
/// Record the serving Tokio runtime's worker and blocking-pool thread counts.
/// Blocking-pool counts require a diagnostic build with `tokio_unstable`.
fn record_tokio_runtime_metrics() {
let runtime = tokio::runtime::Handle::current().metrics();
metrics::gauge!("rustfs_runtime_worker_threads").set(runtime.num_workers() as f64);
#[cfg(tokio_unstable)]
{
let total = runtime.num_blocking_threads();
let idle = runtime.num_idle_blocking_threads().min(total);
metrics::gauge!("rustfs_runtime_blocking_pool_metrics_available").set(1.0);
metrics::gauge!("rustfs_runtime_blocking_threads_total").set(total as f64);
metrics::gauge!("rustfs_runtime_blocking_threads_idle").set(idle as f64);
metrics::gauge!("rustfs_runtime_blocking_threads_active").set(total.saturating_sub(idle) as f64);
}
#[cfg(not(tokio_unstable))]
metrics::gauge!("rustfs_runtime_blocking_pool_metrics_available").set(0.0);
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
{
metrics::gauge!("rustfs_memory_allocator_rust_global_requested_live_bytes_available").set(1.0);
metrics::gauge!("rustfs_memory_allocator_rust_global_requested_live_bytes")
.set(DIAGNOSTIC_GLOBAL_ALLOCATOR_LIVE_REQUESTED_BYTES.load(Ordering::Relaxed) as f64);
}
#[cfg(not(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
)))]
metrics::gauge!("rustfs_memory_allocator_rust_global_requested_live_bytes_available").set(0.0);
}
async fn record_memory_snapshot(process_sampler: Arc<Mutex<ProcessSampler>>) {
record_tokio_runtime_metrics();
match tokio::task::spawn_blocking(move || {
let mut sampler = process_sampler.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let (resource, process) = sampler.snapshot_resource_and_system();
@@ -535,6 +612,35 @@ pub fn init_memory_observability(ctx: CancellationToken) {
#[cfg(test)]
mod tests {
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
use std::sync::atomic::AtomicUsize;
#[cfg(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
))]
#[test]
fn diagnostic_global_allocator_counter_tracks_alloc_realloc_and_free() {
let counter = AtomicUsize::new(0);
super::adjust_diagnostic_global_allocator_live_bytes(&counter, 0, 32);
assert_eq!(counter.load(std::sync::atomic::Ordering::Relaxed), 32);
super::adjust_diagnostic_global_allocator_live_bytes(&counter, 32, 64);
assert_eq!(counter.load(std::sync::atomic::Ordering::Relaxed), 64);
super::adjust_diagnostic_global_allocator_live_bytes(&counter, 64, 8);
assert_eq!(counter.load(std::sync::atomic::Ordering::Relaxed), 8);
super::adjust_diagnostic_global_allocator_live_bytes(&counter, 8, 0);
assert_eq!(counter.load(std::sync::atomic::Ordering::Relaxed), 0);
}
use super::{
CgroupMemorySnapshot, MEMORY_OBSERVABILITY_SERVICE_NAME, MemoryObservabilityCancellationSource,
MemoryObservabilityController, MemoryObservabilityDesiredState, MemoryObservabilityServiceState,
@@ -544,6 +650,45 @@ mod tests {
use std::fs;
use std::path::PathBuf;
#[tokio::test]
async fn runtime_diagnostics_publish_feature_availability() {
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
let recorder = DebuggingRecorder::new();
metrics::with_local_recorder(&recorder, super::record_tokio_runtime_metrics);
let rows = recorder.snapshotter().snapshot().into_vec();
let gauge = |name: &str| {
rows.iter()
.find_map(|(key, _, _, value)| match value {
DebugValue::Gauge(value) if key.key().name() == name => Some(value.into_inner()),
_ => None,
})
.expect("diagnostic gauge is published")
};
assert_eq!(
gauge("rustfs_runtime_worker_threads"),
f64::from(
u32::try_from(tokio::runtime::Handle::current().metrics().num_workers()).expect("Tokio worker count fits u32")
)
);
assert_eq!(
gauge("rustfs_runtime_blocking_pool_metrics_available"),
if cfg!(tokio_unstable) { 1.0 } else { 0.0 }
);
let allocator_available = cfg!(all(
feature = "diagnostic-allocator-live",
not(target_os = "windows"),
not(all(feature = "hotpath", feature = "hotpath-alloc"))
));
assert_eq!(
gauge("rustfs_memory_allocator_rust_global_requested_live_bytes_available"),
if allocator_available { 1.0 } else { 0.0 }
);
if allocator_available {
assert!(gauge("rustfs_memory_allocator_rust_global_requested_live_bytes") >= 0.0);
}
}
#[cfg(unix)]
#[test]
fn live_resource_metrics_follow_shrink_and_preserve_unknown_observations() {
@@ -15,13 +15,18 @@
//! The gateway operations of migration batch B2 (rustfs/backlog#2766): object listings, multipart
//! listings, and multipart create and abort.
use super::{OpRow, Served, row};
use super::{
OpRow, Served,
object_write::{request_version_id, retain_until},
row,
};
use crate::app::RequestEnvelope;
use crate::app::bucket_usecase::DefaultBucketUsecase;
use crate::app::multipart_usecase::DefaultMultipartUsecase;
use crate::app::multipart_usecase::{CreateMultipartUploadFields, DefaultMultipartUsecase};
use crate::runtime_sources::{AppContext, ServerContextSlot};
use crate::storage_api::server::http::access::access_owned::listing_multipart::{
authorize_bucket_listing, authorize_list_multipart_uploads, authorize_list_objects_v2,
CreateMultipartAuthorization, authorize_bucket_listing, authorize_create_multipart_upload, authorize_list_multipart_uploads,
authorize_list_objects_v2,
};
use rustfs_gateway::{Req, Resp, dto};
use rustfs_policy::policy::action::S3Action;
@@ -34,6 +39,7 @@ pub(super) const ROWS: &[OpRow] = &[
row::<dto::ListObjects>(),
row::<dto::ListObjectVersions>(),
row::<dto::ListMultipartUploads>(),
row::<dto::CreateMultipartUpload>(),
];
impl Served for dto::ListObjectsV2 {
@@ -98,6 +104,60 @@ impl Served for dto::ListMultipartUploads {
}
}
impl Served for dto::CreateMultipartUpload {
const METRIC: S3Operation = S3Operation::CreateMultipartUpload;
async fn authorize(envelope: &mut RequestEnvelope, request: &Req<Self>) -> S3Result<()> {
let server = server_context(envelope)?;
let input = request.input();
let version = request_version_id(envelope);
let facts = CreateMultipartAuthorization {
bucket: input.bucket.as_str(),
key: input.key.as_str(),
query_version: version.as_deref(),
retain_until: retain_until(input.object_lock_retain_until_date.as_ref())?,
legal_hold_present: input.object_lock_legal_hold_status.is_some(),
retention_present: input.object_lock_mode.is_some() || input.object_lock_retain_until_date.is_some(),
};
authorize_create_multipart_upload(&server, envelope, facts).await
}
async fn execute(context: Option<Arc<AppContext>>, envelope: RequestEnvelope, request: Req<Self>) -> S3Result<Resp<Self>> {
let version = request_version_id(&envelope);
let input = request.into_input();
let fields = CreateMultipartUploadFields {
bucket: input.bucket.as_str().to_owned(),
key: input.key.as_str().to_owned(),
metadata: Some(input.metadata.into_iter().collect()),
tagging: input.tagging,
storage_class: input.storage_class.map(|value| value.as_str().to_owned()),
server_side_encryption: input.server_side_encryption.map(|value| value.as_str().to_owned()),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key.map(|value| value.expose_secret().to_owned()),
sse_customer_key_md5: input.sse_customer_key_md5,
ssekms_key_id: input.ssekms_key_id,
object_lock_legal_hold_status: input.object_lock_legal_hold_status.map(|value| value.as_str().to_owned()),
object_lock_mode: input.object_lock_mode.map(|value| value.as_str().to_owned()),
object_lock_retain_until_date: retain_until(input.object_lock_retain_until_date.as_ref())?,
};
let output = DefaultMultipartUsecase::with_context(context)
.execute_create_multipart_upload(envelope, fields, version)
.await?;
Ok(Resp::new(dto::CreateMultipartUploadOutput {
bucket: input.bucket,
key: input.key,
upload_id: output.upload_id,
server_side_encryption: output.server_side_encryption.map(dto::ServerSideEncryption::custom),
sse_customer_algorithm: output.sse_customer_algorithm,
sse_customer_key_md5: output.sse_customer_key_md5,
ssekms_key_id: output.ssekms_key_id,
checksum_algorithm: output.checksum_algorithm.map(dto::ChecksumAlgorithm::custom),
checksum_type: output.checksum_type.map(dto::ChecksumType::custom),
..Default::default()
}))
}
}
/// The server context a generation guard is read through; the gateway envelope always has one.
fn server_context(envelope: &RequestEnvelope) -> S3Result<Arc<ServerContextSlot>> {
envelope
@@ -16,16 +16,20 @@
use super::{OpRow, Served, row};
use crate::app::RequestEnvelope;
use crate::app::multipart_usecase::{DefaultMultipartUsecase, UploadPartTarget, UploadPartWriteInput, UploadPartWriteOutput};
use crate::app::object::FrameTrailers;
use crate::app::object_usecase::{DefaultObjectUsecase, PutObjectWriteInput, PutObjectWriteOutput};
use crate::runtime_sources::AppContext;
use crate::storage_api::server::http::access::access_owned::object_write::{PutObjectAuthorization, authorize_put_object};
use crate::storage_api::server::http::access::access_owned::object_write::{
PutObjectAuthorization, authorize_put_object, authorize_upload_part,
};
use rustfs_gateway::{Req, Resp, dto};
use rustfs_gateway_types::{ChecksumAlgorithm, Timestamp};
use rustfs_s3_ops::S3Operation;
use rustfs_s3_types::{S3Result, s3_error};
use std::sync::Arc;
pub(super) const ROWS: &[OpRow] = &[row::<dto::PutObject>()];
pub(super) const ROWS: &[OpRow] = &[row::<dto::PutObject>(), row::<dto::UploadPart>()];
impl Served for dto::PutObject {
const METRIC: S3Operation = S3Operation::PutObject;
@@ -61,13 +65,7 @@ impl Served for dto::PutObject {
async fn execute(context: Option<Arc<AppContext>>, envelope: RequestEnvelope, request: Req<Self>) -> S3Result<Resp<Self>> {
let mut input = native_input(request.into_input())?;
input.version_id = request_version_id(&envelope);
if let Some(declaration) = envelope.headers().get("x-amz-trailer").filter(|value| !value.is_empty()) {
// The gateway verifies the declared trailer independently of the SDK algorithm.
let algorithm = declaration
.to_str()
.ok()
.and_then(|name| ChecksumAlgorithm::from_header_name(name.trim_matches([' ', '\t'])))
.ok_or_else(|| s3_error!(InvalidRequest, "Invalid checksum trailer declaration"))?;
if let Some(algorithm) = trailer_checksum_algorithm(&envelope)? {
input.checksum_algorithm = Some(algorithm.wire_name().to_owned());
input.capture_frame_trailers();
}
@@ -78,14 +76,118 @@ impl Served for dto::PutObject {
}
}
// PutObject's destination version is a RustFS replication extension, absent from the AWS DTO.
fn request_version_id(envelope: &RequestEnvelope) -> Option<String> {
impl Served for dto::UploadPart {
const METRIC: S3Operation = S3Operation::UploadPart;
fn authorize(envelope: &mut RequestEnvelope, request: &Req<Self>) -> impl std::future::Future<Output = S3Result<()>> + Send {
let input = request.input();
let bucket = input.bucket.as_str().to_owned();
let key = input.key.as_str().to_owned();
async move {
let server = envelope
.server_context()
.cloned()
.ok_or_else(|| s3_error!(InternalError, "server context is not initialized"))?;
Ok(authorize_upload_part(envelope, &server, &bucket, &key).await?)
}
}
async fn execute(context: Option<Arc<AppContext>>, envelope: RequestEnvelope, request: Req<Self>) -> S3Result<Resp<Self>> {
let input = native_part_input(request.into_input(), trailer_checksum_algorithm(&envelope)?);
let output = DefaultMultipartUsecase::with_context(context)
.execute_upload_part_owned(envelope, input)
.await?;
native_part_output(output)
}
}
fn trailer_checksum_algorithm(envelope: &RequestEnvelope) -> S3Result<Option<ChecksumAlgorithm>> {
envelope
.headers()
.get("x-amz-trailer")
.filter(|value| !value.is_empty())
.map(|declaration| {
// The gateway verifies the declared trailer independently of the SDK algorithm.
declaration
.to_str()
.ok()
.and_then(|name| ChecksumAlgorithm::from_header_name(name.trim_matches([' ', '\t'])))
.ok_or_else(|| s3_error!(InvalidRequest, "Invalid checksum trailer declaration"))
})
.transpose()
}
fn native_part_input(input: dto::UploadPartInput, trailer_algorithm: Option<ChecksumAlgorithm>) -> UploadPartWriteInput {
let checksum = |algorithm| {
input
.checksum_spec
.as_ref()
.filter(|value| value.algorithm() == algorithm)
.map(|value| value.render_base64().to_owned())
};
let frame_trailers = trailer_algorithm.map(|_| Arc::new(FrameTrailers::default()));
UploadPartWriteInput {
checksum_crc32: checksum(ChecksumAlgorithm::Crc32),
checksum_crc32c: checksum(ChecksumAlgorithm::Crc32c),
checksum_crc64nvme: checksum(ChecksumAlgorithm::Crc64Nvme),
checksum_sha1: checksum(ChecksumAlgorithm::Sha1),
checksum_sha256: checksum(ChecksumAlgorithm::Sha256),
target: UploadPartTarget::Gateway {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
content_length: Some(input.content_length),
body: input.body.map(|body| body.into_body()),
trailing_headers: frame_trailers.clone().map(|source| source as rustfs_rio::SharedTrailerSource),
frame_trailers,
content_md5: input.content_md5,
checksum_algorithm: trailer_algorithm
.map(|value| value.wire_name().to_owned())
.or_else(|| input.checksum_algorithm.map(|value| value.as_str().to_owned())),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key.map(|value| value.expose_secret().to_owned()),
sse_customer_key_md5: input.sse_customer_key_md5,
}
}
fn native_part_output(output: UploadPartWriteOutput) -> S3Result<Resp<dto::UploadPart>> {
let mut headers = output.headers;
insert_checksum_headers(
&mut headers,
[
output.checksum_crc32,
output.checksum_crc32c,
output.checksum_sha1,
output.checksum_sha256,
output.checksum_crc64nvme,
],
)?;
let dto = dto::UploadPartOutput {
e_tag: crate::app::dto_values::stored_etag(
output
.e_tag
.as_deref()
.ok_or_else(|| s3_error!(InternalError, "UploadPart completed without an ETag"))?,
),
server_side_encryption: output.server_side_encryption.map(dto::ServerSideEncryption::custom),
sse_customer_algorithm: output.sse_customer_algorithm,
sse_customer_key_md5: output.sse_customer_key_md5,
ssekms_key_id: output.ssekms_key_id,
..Default::default()
};
Ok(Resp::new(dto).with_extra_headers(headers))
}
// The destination version is a RustFS replication extension absent from the AWS DTOs.
pub(super) fn request_version_id(envelope: &RequestEnvelope) -> Option<String> {
url::form_urlencoded::parse(envelope.uri().query()?.as_bytes())
.find(|(name, _)| name == "versionId")
.map(|(_, value)| value.into_owned())
}
fn retain_until(value: Option<&Timestamp>) -> S3Result<Option<time::OffsetDateTime>> {
pub(super) fn retain_until(value: Option<&Timestamp>) -> S3Result<Option<time::OffsetDateTime>> {
value
.map(|value| {
time::OffsetDateTime::from_unix_timestamp_nanos(
@@ -143,22 +245,16 @@ fn native_input(input: dto::PutObjectInput) -> S3Result<PutObjectWriteInput> {
fn native_output(output: PutObjectWriteOutput) -> S3Result<Resp<dto::PutObject>> {
let mut headers = output.headers;
for (name, value) in [
("x-amz-checksum-crc32", output.checksum_crc32),
("x-amz-checksum-crc32c", output.checksum_crc32c),
("x-amz-checksum-sha1", output.checksum_sha1),
("x-amz-checksum-sha256", output.checksum_sha256),
("x-amz-checksum-crc64nvme", output.checksum_crc64nvme),
] {
if let Some(value) = value {
headers.insert(
name,
value
.parse()
.map_err(|_| s3_error!(InternalError, "Invalid stored checksum"))?,
);
}
}
insert_checksum_headers(
&mut headers,
[
output.checksum_crc32,
output.checksum_crc32c,
output.checksum_sha1,
output.checksum_sha256,
output.checksum_crc64nvme,
],
)?;
let dto = dto::PutObjectOutput {
e_tag: crate::app::dto_values::stored_etag(
output
@@ -181,6 +277,29 @@ fn native_output(output: PutObjectWriteOutput) -> S3Result<Resp<dto::PutObject>>
.with_extra_headers(headers))
}
fn insert_checksum_headers(
headers: &mut http::HeaderMap,
[crc32, crc32c, sha1, sha256, crc64nvme]: [Option<String>; 5],
) -> S3Result<()> {
for (name, value) in [
("x-amz-checksum-crc32", crc32),
("x-amz-checksum-crc32c", crc32c),
("x-amz-checksum-sha1", sha1),
("x-amz-checksum-sha256", sha256),
("x-amz-checksum-crc64nvme", crc64nvme),
] {
if let Some(value) = value {
headers.insert(
name,
value
.parse()
.map_err(|_| s3_error!(InternalError, "Invalid stored checksum"))?,
);
}
}
Ok(())
}
#[cfg(test)]
#[path = "object_write_test.rs"]
mod tests;
@@ -17,9 +17,9 @@
use super::*;
use crate::runtime_sources::ServerContextSlot;
use crate::storage_api::server::http::access::test_support::{
AuthorizationRequest, BucketOperations as _, ETag, ETagCondition, FS, MakeBucketOptions, ObjectOperations as _,
ObjectOptions, PutObjectInput as LegacyPutObjectInput, ReqInfo, S3Access as _, S3Request, get_bucket_metadata,
set_bucket_metadata,
AuthorizationRequest, BucketOperations as _, CompletePart, DeleteBucketOptions, ETag, ETagCondition, FS, MakeBucketOptions,
MultipartOperations as _, ObjectIO as _, ObjectOperations as _, ObjectOptions, PutObjectInput as LegacyPutObjectInput,
ReqInfo, S3Access as _, S3Request, get_bucket_metadata, set_bucket_metadata,
};
use bytes::Bytes;
use http::{HeaderMap, HeaderValue, Method};
@@ -30,7 +30,7 @@ use std::collections::VecDeque;
use std::pin::Pin;
use std::task::{Context, Poll};
fn request(input: dto::PutObjectInput) -> Req<dto::PutObject> {
fn request<O: rustfs_gateway::Operation<DerivedResources = rustfs_gateway::NoDerived>>(input: O::Input) -> Req<O> {
let wire = WireRequest::accept(
http::Request::builder()
.uri("/")
@@ -466,3 +466,181 @@ fn native_put_response_refuses_a_missing_etag() {
};
assert_eq!(error.code().as_str(), "InternalError");
}
fn part_input(bucket: &str, upload_id: &str, part_number: i32) -> dto::UploadPartInput {
dto::UploadPartInput {
bucket: rustfs_gateway_types::BucketName::new(bucket).unwrap(),
key: rustfs_gateway_types::ObjectKey::new("allowed").unwrap(),
upload_id: rustfs_gateway_types::UploadIdClaim::from_wire(upload_id),
part_number,
content_length: 3,
body: Some(ByteStream::from_bytes(Bytes::from_static(b"abc"))),
..Default::default()
}
}
#[tokio::test]
#[serial]
async fn native_upload_part_rejects_missing_and_stale_generation() {
let (server, bucket) = fixture(&["s3:PutObject"]).await;
let store = server.object_store().unwrap();
let options = ObjectOptions::default();
let key = "allowed";
// The legitimate authorization captures an incarnation while the request is still being admitted.
let mut authorized = envelope(&server, HeaderMap::new());
let request_without_session = request::<dto::UploadPart>(part_input(&bucket, "not-yet-active", 1));
<dto::UploadPart as Served>::authorize(&mut authorized, &request_without_session)
.await
.unwrap();
let failed =
<dto::UploadPart as Served>::execute(server.app_context(), authorized.fork_for_authorization(), request_without_session)
.await
.err()
.unwrap();
assert_eq!(failed.code().as_str(), "NoSuchUpload");
store.delete_bucket(&bucket, &DeleteBucketOptions::default()).await.unwrap();
store.make_bucket(&bucket, &MakeBucketOptions::default()).await.unwrap();
let session = store.new_multipart_upload(&bucket, key, &options).await.unwrap().upload_id;
let missing = envelope(&server, HeaderMap::new());
for (envelope, code) in [(missing, "InternalError"), (authorized, "NoSuchBucket")] {
let request = request::<dto::UploadPart>(part_input(&bucket, &session, 1));
let error = <dto::UploadPart as Served>::execute(server.app_context(), envelope, request)
.await
.err()
.unwrap();
assert_eq!(error.code().as_str(), code);
assert!(
store
.list_object_parts(&bucket, key, &session, None, 20, &options)
.await
.unwrap()
.parts
.is_empty()
);
}
// The identical part is writable under a freshly captured generation.
set_policy(
&bucket,
serde_json::json!({
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": "s3:PutObject",
"Resource": format!("arn:aws:s3:::{bucket}/{key}"),
}),
)
.await;
let req = request::<dto::UploadPart>(part_input(&bucket, &session, 1));
let mut fresh = envelope(&server, HeaderMap::new());
<dto::UploadPart as Served>::authorize(&mut fresh, &req).await.unwrap();
<dto::UploadPart as Served>::execute(server.app_context(), fresh, req)
.await
.unwrap();
let parts = store
.list_object_parts(&bucket, key, &session, None, 20, &options)
.await
.unwrap()
.parts;
assert_eq!(
parts.first().and_then(|part| part.etag.as_deref()),
Some("900150983cd24fb0d6963f7d28e17f72")
);
}
#[tokio::test]
#[serial]
async fn native_upload_part_uses_frames_and_leaves_the_prior_part_on_failure() {
use tokio::io::AsyncReadExt;
let (server, bucket) = fixture(&["s3:PutObject"]).await;
let store = server.object_store().unwrap();
let options = ObjectOptions::default();
let key = "allowed";
let upload_id = store.new_multipart_upload(&bucket, key, &options).await.unwrap().upload_id;
let etag = "900150983cd24fb0d6963f7d28e17f72";
for (label, payload, size, checksum, expected) in [
("valid", b"abc", 3, Some("NSRBwg=="), None),
("missing trailer", b"xyz", 3, None, Some("BadDigest")),
("wrong checksum", b"xyz", 3, Some("AAAAAA=="), Some("BadDigest")),
("short", b"xyz", 4, Some("6466Zw=="), Some("IncompleteBody")),
("digest error", b"xyz", 3, None, Some("BadDigest")),
("signature error", b"xyz", 3, None, Some("SignatureDoesNotMatch")),
] {
let mut headers = HeaderMap::new();
headers.insert("x-amz-trailer", HeaderValue::from_static("x-amz-checksum-crc32"));
let mut trailers = HeaderMap::new();
if let Some(value) = checksum {
trailers.insert("x-amz-checksum-crc32", HeaderValue::from_static(value));
}
let end = match label {
"digest error" => Err(StreamError::upstream(Box::new(rustfs_rio::BadDigest {
expected_md5: String::new(),
calculated_md5: String::new(),
}))),
"signature error" => Err(StreamError::upstream(Box::new(
crate::app::storage_api::s3::stream::aws_chunked_stream::AwsChunkedStreamError::SignatureMismatch,
))),
_ => Ok(PayloadRead::Eof {
trailers: TrailingHeaders::from_header_map(trailers),
}),
};
let mut input = part_input(&bucket, &upload_id, 1);
input.content_length = size;
input.body = Some(
ByteStream::new(Box::pin(Frames(VecDeque::from([
Ok(PayloadRead::Chunk(Bytes::from_static(payload))),
end,
]))))
.unwrap(),
);
// The decoded trailer declaration wins even with another SDK algorithm value.
input.checksum_algorithm = Some(dto::ChecksumAlgorithm::custom("SHA256"));
let request = request::<dto::UploadPart>(input);
let mut context = envelope(&server, headers);
<dto::UploadPart as Served>::authorize(&mut context, &request).await.unwrap();
let result = <dto::UploadPart as Served>::execute(server.app_context(), context, request).await;
assert_eq!(result.as_ref().err().map(|error| error.code().as_str()), expected, "{label}");
if let Ok(response) = result {
assert_eq!(response.output().unwrap().e_tag, crate::app::dto_values::stored_etag(etag));
let (_, status, headers) = response.into_parts();
assert_eq!(status, 200);
assert_eq!(headers["x-amz-checksum-crc32"], "NSRBwg==");
}
let parts = store
.list_object_parts(&bucket, key, &upload_id, None, 20, &options)
.await
.unwrap()
.parts;
assert_eq!(parts.len(), 1, "{label}");
assert_eq!(
(parts[0].part_num, parts[0].size, parts[0].etag.as_deref()),
(1, 3, Some(etag)),
"{label}"
);
}
store
.clone()
.complete_multipart_upload(
&bucket,
key,
&upload_id,
vec![CompletePart {
part_num: 1,
etag: Some(etag.to_owned()),
..Default::default()
}],
&options,
)
.await
.unwrap();
let mut old = store
.get_object_reader(&bucket, key, None, HeaderMap::new(), &options)
.await
.unwrap();
let mut bytes = Vec::new();
old.stream.read_to_end(&mut bytes).await.unwrap();
assert_eq!(bytes, b"abc");
}
+373 -3
View File
@@ -20,7 +20,11 @@ use rustfs_config::{
use rustfs_utils::os::DiskInfo;
use std::collections::BTreeSet;
use std::io::{Error, Result};
use tracing::warn;
use tracing::{info, warn};
const LOG_COMPONENT_STORAGE: &str = "storage";
const LOG_SUBSYSTEM_STARTUP: &str = "startup";
const EVENT_STARTUP_FS_BOUNDARY: &str = "startup_fs_boundary";
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
enum UnsupportedFsPolicy {
@@ -61,6 +65,209 @@ fn is_unsupported_fs_type(fs_type: &str) -> bool {
) || normalized.starts_with("fuse.")
}
fn is_ninep_fs_type(fs_type: &str) -> bool {
let normalized = fs_type.trim().to_ascii_lowercase();
matches!(normalized.as_str(), "9p" | "v9fs")
}
/// One `/proc/self/mountinfo` record.
///
/// Fields are `(mount id) (parent) (major:minor) (root) (mount point) (options)
/// (optional fields) - (fs type) (source) (super options)`.
struct MountInfoRecord {
mount_point: String,
fs_type: String,
options: String,
}
/// gVisor (runsc) directfs bind mounts are host POSIX files opened through
/// donated file descriptors. The sandbox still reports them as 9p, and the
/// mountinfo options include both `trans=fd` and `directfs`. Real 9p and
/// virtio shares do not carry that pair, so the host type stays hidden only
/// for this shape.
fn is_gvisor_directfs_bind_mount(fs_type: &str, mountinfo: &str, path: &str) -> bool {
if !is_ninep_fs_type(fs_type) {
return false;
}
let records = parse_mountinfo(mountinfo);
let Some(record) = covering_mount(&records, path) else {
return false;
};
is_ninep_fs_type(&record.fs_type) && options_include_gvisor_directfs(&record.options)
}
fn options_include_gvisor_directfs(options: &str) -> bool {
let mut has_trans_fd = false;
let mut has_directfs = false;
for option in options.split(',') {
match option.trim() {
"trans=fd" => has_trans_fd = true,
"directfs" => has_directfs = true,
_ => {}
}
if has_trans_fd && has_directfs {
return true;
}
}
false
}
fn parse_mountinfo(mountinfo: &str) -> Vec<MountInfoRecord> {
mountinfo.lines().filter_map(parse_mountinfo_line).collect()
}
fn parse_mountinfo_line(line: &str) -> Option<MountInfoRecord> {
let line = line.trim();
if line.is_empty() {
return None;
}
let mut fields = line.split_whitespace();
// mount id, parent id, major:minor, root
for _ in 0..4 {
fields.next()?;
}
let mount_point = normalize_mount_point(unescape_mountinfo_path(fields.next()?));
let mount_options = fields.next()?;
let mut seen_separator = false;
let mut after_separator = 0usize;
let mut fs_type = None;
let mut super_options = None;
for field in fields {
if !seen_separator {
if field == "-" {
seen_separator = true;
}
continue;
}
after_separator += 1;
match after_separator {
1 => fs_type = Some(field),
2 => {}
3 => {
super_options = Some(field);
break;
}
_ => break,
}
}
let fs_type = fs_type?;
let super_options = super_options?;
Some(MountInfoRecord {
mount_point,
fs_type: fs_type.to_string(),
// Mount options and super options are both "mountinfo options".
// gVisor puts `trans=fd` and `directfs` in the super options.
options: format!("{mount_options},{super_options}"),
})
}
fn unescape_mountinfo_path(raw: &str) -> String {
if !raw.contains('\\') {
return raw.to_string();
}
let mut out = String::with_capacity(raw.len());
let mut rest = raw;
while let Some(offset) = rest.find('\\') {
out.push_str(&rest[..offset]);
let after = &rest[offset + 1..];
let octal_digits = after.len() >= 3 && after.as_bytes()[..3].iter().all(|digit| (b'0'..=b'7').contains(digit));
if octal_digits {
match u8::from_str_radix(&after[..3], 8) {
Ok(value) => {
out.push(char::from(value));
rest = &after[3..];
}
Err(_) => {
out.push('\\');
rest = after;
}
}
} else {
out.push('\\');
rest = after;
}
}
out.push_str(rest);
out
}
fn normalize_mount_point(mut mount_point: String) -> String {
if mount_point.len() > 1 {
while mount_point.ends_with('/') {
mount_point.pop();
}
}
mount_point
}
fn trim_trailing_slash(path: &str) -> &str {
if path.len() > 1 && path.ends_with('/') {
path.trim_end_matches('/')
} else {
path
}
}
fn mount_covers(mount_point: &str, path: &str) -> bool {
if mount_point == "/" {
return path.starts_with('/');
}
path == mount_point || path.starts_with(&format!("{mount_point}/"))
}
/// The visible mount is the longest mount point that contains `path`.
/// A later record of the same length wins, matching mount order in mountinfo.
fn covering_mount<'a>(records: &'a [MountInfoRecord], path: &str) -> Option<&'a MountInfoRecord> {
let path = trim_trailing_slash(path);
let mut best: Option<&MountInfoRecord> = None;
for record in records {
if !mount_covers(&record.mount_point, path) {
continue;
}
let replaces_best = best.is_none_or(|current| record.mount_point.len() >= current.mount_point.len());
if replaces_best {
best = Some(record);
}
}
best
}
#[derive(Debug, Eq, PartialEq)]
enum EndpointFsClass {
Supported,
GvisorDirectfs,
Unsupported(String),
}
fn classify_endpoint_fs(info: &DiskInfo, mountinfo: &str, path: &str) -> EndpointFsClass {
// A Windows network redirector stays unsupported even when the reported
// filesystem name would otherwise pass. gVisor directfs is not that case.
if !info.remote && is_gvisor_directfs_bind_mount(&info.fstype, mountinfo, path) {
return EndpointFsClass::GvisorDirectfs;
}
match unsupported_fs_label(info) {
Some(label) => EndpointFsClass::Unsupported(label),
None => EndpointFsClass::Supported,
}
}
fn read_self_mountinfo() -> String {
#[cfg(target_os = "linux")]
{
// An unreadable mount table cannot prove directfs. Leaving it empty
// keeps a 9p/V9FS endpoint on the unsupported path.
std::fs::read_to_string("/proc/self/mountinfo").unwrap_or_default()
}
#[cfg(not(target_os = "linux"))]
{
String::new()
}
}
/// Describes why the filesystem behind `info` is outside the supported
/// boundary, or returns `None` when it is supported.
fn unsupported_fs_label(info: &DiskInfo) -> Option<String> {
@@ -90,7 +297,9 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools
return Ok(());
}
let mountinfo = read_self_mountinfo();
let mut unsupported = Vec::new();
let mut gvisor_directfs = Vec::new();
for path in local_paths {
let info = match rustfs_utils::os::get_info(&path) {
Ok(info) => info,
@@ -104,11 +313,25 @@ pub(crate) fn enforce_unsupported_fs_policy(endpoint_pools: &EndpointServerPools
}
};
if let Some(label) = unsupported_fs_label(&info) {
unsupported.push((path, label));
match classify_endpoint_fs(&info, &mountinfo, &path) {
EndpointFsClass::Supported => {}
EndpointFsClass::GvisorDirectfs => gvisor_directfs.push(path),
EndpointFsClass::Unsupported(label) => unsupported.push((path, label)),
}
}
if !gvisor_directfs.is_empty() {
let path = gvisor_directfs.join(", ");
info!(
event = EVENT_STARTUP_FS_BOUNDARY,
component = LOG_COMPONENT_STORAGE,
subsystem = LOG_SUBSYSTEM_STARTUP,
state = "gvisor_directfs",
path = %path,
"Host filesystem type is not visible through the sandbox (gVisor directfs bind mount)"
);
}
if unsupported.is_empty() {
return Ok(());
}
@@ -202,4 +425,151 @@ mod tests {
);
});
}
/// Reporter mountinfo from a gVisor directfs bind of a host directory on `/data`.
const GVISOR_DIRECTFS_MOUNTINFO: &str = "\
23 18 0:26 / /data rw - 9p none rw,trans=fd,rfdno=6,wfdno=6,aname=/,dfltuid=4294967294,dfltgid=4294967294,dcache=1000,cache=remote_revalidating,disable_fifo_open,directfs
";
const VIRTIO_9P_MOUNTINFO: &str = "\
36 22 0:40 / /mnt/virtio rw,relatime shared:1 - 9p hostshare rw,trans=virtio,version=9p2000.L,posixacl,msize=104857600
";
const TCP_9P_MOUNTINFO: &str = "\
41 22 0:41 / /mnt/tcp rw,relatime - 9p 10.0.0.8 rw,trans=tcp,version=9p2000.L
";
fn v9fs_disk() -> DiskInfo {
DiskInfo {
fstype: "V9FS".to_string(),
..Default::default()
}
}
#[test]
fn gvisor_directfs_bind_mount_is_not_unsupported_v9fs() {
let info = v9fs_disk();
assert_eq!(
classify_endpoint_fs(&info, GVISOR_DIRECTFS_MOUNTINFO, "/data"),
EndpointFsClass::GvisorDirectfs
);
assert!(is_gvisor_directfs_bind_mount("v9fs", GVISOR_DIRECTFS_MOUNTINFO, "/data"));
assert!(is_gvisor_directfs_bind_mount("9p", GVISOR_DIRECTFS_MOUNTINFO, "/data/"));
// The endpoint may be a directory inside the bind mount.
assert_eq!(
classify_endpoint_fs(&info, GVISOR_DIRECTFS_MOUNTINFO, "/data/rustfs"),
EndpointFsClass::GvisorDirectfs
);
}
#[test]
fn plain_9p_and_virtio_mounts_stay_unsupported() {
let info = v9fs_disk();
assert_eq!(
classify_endpoint_fs(&info, VIRTIO_9P_MOUNTINFO, "/mnt/virtio"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
assert_eq!(
classify_endpoint_fs(&info, TCP_9P_MOUNTINFO, "/mnt/tcp"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
assert_eq!(classify_endpoint_fs(&info, "", "/data"), EndpointFsClass::Unsupported("V9FS".to_string()));
assert!(is_unsupported_fs_type("v9fs"));
assert!(is_unsupported_fs_type("9p"));
}
#[test]
fn gvisor_exception_requires_both_trans_fd_and_directfs() {
let info = v9fs_disk();
let trans_fd_only = "23 18 0:26 / /data rw - 9p none rw,trans=fd,rfdno=6,wfdno=6,aname=/";
let directfs_only = "23 18 0:26 / /data rw - 9p none rw,directfs,aname=/";
let trans_fd_prefix = "23 18 0:26 / /data rw - 9p none rw,trans=fdx,directfs";
let directfs_substring = "23 18 0:26 / /data rw - 9p none rw,trans=virtio,cache=directfs";
for mountinfo in [trans_fd_only, directfs_only, trans_fd_prefix, directfs_substring] {
assert_eq!(
classify_endpoint_fs(&info, mountinfo, "/data"),
EndpointFsClass::Unsupported("V9FS".to_string()),
"mountinfo should stay unsupported: {mountinfo}"
);
}
}
#[test]
fn gvisor_exception_does_not_apply_to_other_filesystem_types() {
for fstype in ["NFS", "CIFS", "FUSE", "fuse.sshfs", "overlayfs"] {
let info = DiskInfo {
fstype: fstype.to_string(),
..Default::default()
};
assert_eq!(
classify_endpoint_fs(&info, GVISOR_DIRECTFS_MOUNTINFO, "/data"),
EndpointFsClass::Unsupported(fstype.to_string())
);
}
// statfs says V9FS, but the covering mount is not 9p. Do not exempt it.
let nfs_with_same_options = "50 1 0:50 / /data rw - nfs 10.0.0.1:/export rw,trans=fd,directfs";
assert_eq!(
classify_endpoint_fs(&v9fs_disk(), nfs_with_same_options, "/data"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
let xfs = DiskInfo {
fstype: "XFS".to_string(),
..Default::default()
};
assert_eq!(classify_endpoint_fs(&xfs, GVISOR_DIRECTFS_MOUNTINFO, "/data"), EndpointFsClass::Supported);
}
#[test]
fn gvisor_exception_uses_the_covering_mount() {
let mixed = "\
18 1 8:1 / / rw,relatime - ext4 /dev/sda1 rw
36 18 0:40 / /mnt/virtio rw,relatime shared:1 - 9p hostshare rw,trans=virtio,version=9p2000.L
23 18 0:26 / /data rw - 9p none rw,trans=fd,rfdno=6,wfdno=6,aname=/,dfltuid=4294967294,dfltgid=4294967294,dcache=1000,cache=remote_revalidating,disable_fifo_open,directfs
41 18 0:41 / /mnt/tcp rw - 9p 10.0.0.8 rw,trans=tcp,version=9p2000.L
";
let info = v9fs_disk();
assert_eq!(classify_endpoint_fs(&info, mixed, "/data"), EndpointFsClass::GvisorDirectfs);
assert_eq!(
classify_endpoint_fs(&info, mixed, "/mnt/virtio"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
assert_eq!(
classify_endpoint_fs(&info, mixed, "/mnt/tcp"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
// `/data` must not exempt the sibling `/database`, which sits on the root mount.
assert_eq!(
classify_endpoint_fs(&info, mixed, "/database"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
}
#[test]
fn escaped_mount_point_matches_the_endpoint_path() {
let mountinfo = "23 18 0:26 / /data\\040disk rw - 9p none rw,trans=fd,directfs";
assert_eq!(
classify_endpoint_fs(&v9fs_disk(), mountinfo, "/data disk"),
EndpointFsClass::GvisorDirectfs
);
assert_eq!(
classify_endpoint_fs(&v9fs_disk(), mountinfo, "/data disk2"),
EndpointFsClass::Unsupported("V9FS".to_string())
);
}
#[test]
fn remote_v9fs_stays_a_network_share() {
let share = DiskInfo {
fstype: "V9FS".to_string(),
remote: true,
..Default::default()
};
assert_eq!(
classify_endpoint_fs(&share, GVISOR_DIRECTFS_MOUNTINFO, "/data"),
EndpointFsClass::Unsupported("V9FS on network share".to_string())
);
}
}
+16 -29
View File
@@ -3441,28 +3441,21 @@ impl S3Access for FS {
/// Checks whether the CreateMultipartUpload request has accesses to the resources.
async fn create_multipart_upload(&self, req: &mut S3Request<CreateMultipartUploadInput>) -> S3Result<()> {
use access_owned::listing_multipart::{CreateMultipartAuthorization, authorize_create_multipart_upload};
let bucket = req.input.bucket.clone();
let bucket_generation = load_bucket_generation(self.server_ctx(), req, &bucket).await;
let req_info = ext_req_info_mut(&mut req.extensions)?;
req_info.bucket = Some(req.input.bucket.clone());
req_info.object = Some(req.input.key.clone());
req_info.version_id = super::options::destination_version_id(req.input.version_id.as_deref(), &req.headers);
let requested_retain_until = req.input.object_lock_retain_until_date.clone();
set_requested_object_lock_retain_until(req, requested_retain_until.map(OffsetDateTime::from));
authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?;
authorize_replication_only_put_headers(req).await?;
if legal_hold_write_requested(req.input.object_lock_legal_hold_status.as_ref()) {
authorize_request(req, Action::S3Action(S3Action::PutObjectLegalHoldAction)).await?;
}
if retention_write_requested(req.input.object_lock_mode.as_ref(), req.input.object_lock_retain_until_date.as_ref()) {
authorize_request(req, Action::S3Action(S3Action::PutObjectRetentionAction)).await?;
}
req.install_bucket_generation(bucket_generation?);
let key = req.input.key.clone();
let query_version = req.input.version_id.clone();
let retain_until = req.input.object_lock_retain_until_date.clone().map(OffsetDateTime::from);
let facts = CreateMultipartAuthorization {
bucket: &bucket,
key: &key,
query_version: query_version.as_deref(),
retain_until,
legal_hold_present: req.input.object_lock_legal_hold_status.is_some(),
retention_present: req.input.object_lock_mode.is_some() || retain_until.is_some(),
};
authorize_create_multipart_upload(self.server_ctx(), req, facts).await?;
Ok(())
}
@@ -4393,14 +4386,8 @@ impl S3Access for FS {
/// This method returns `Ok(())` by default.
async fn upload_part(&self, req: &mut S3Request<UploadPartInput>) -> S3Result<()> {
let bucket = req.input.bucket.clone();
let bucket_generation = load_bucket_generation(self.server_ctx(), req, &bucket).await;
let req_info = ext_req_info_mut(&mut req.extensions)?;
req_info.bucket = Some(req.input.bucket.clone());
req_info.object = Some(req.input.key.clone());
authorize_staged_multipart_part(req).await?;
req.install_bucket_generation(bucket_generation?);
Ok(())
let key = req.input.key.clone();
access_owned::object_write::authorize_upload_part(req, self.server_ctx(), &bucket, &key).await
}
/// Checks whether the UploadPartCopy request has accesses to the resources.
@@ -23,6 +23,7 @@
// The table snapshot reads the access module's private table-catalog state.
use super::super::*;
use crate::storage::options::destination_version_id;
use rustfs_s3_types::S3Result;
use rustfs_utils::http::SUFFIX_INCLUDE_DELETED;
@@ -73,6 +74,43 @@ pub(crate) async fn authorize_list_multipart_uploads<R: AuthorizationRequest + ?
Ok(())
}
/// Decoded request facts used by both multipart initiation access hooks.
pub(crate) struct CreateMultipartAuthorization<'a> {
pub(crate) bucket: &'a str,
pub(crate) key: &'a str,
pub(crate) query_version: Option<&'a str>,
pub(crate) retain_until: Option<OffsetDateTime>,
pub(crate) legal_hold_present: bool,
pub(crate) retention_present: bool,
}
/// Binds authorization to the requested object and captures its bucket incarnation before
/// checking write, replication and Object Lock permissions.
pub(crate) async fn authorize_create_multipart_upload<R: AuthorizationRequest + ?Sized>(
server_ctx: &ServerContextSlot,
req: &mut R,
facts: CreateMultipartAuthorization<'_>,
) -> S3Result<()> {
let bucket_generation = load_bucket_generation(server_ctx, req, facts.bucket).await;
let version = destination_version_id(facts.query_version, req.headers());
let req_info = req_info_mut(req)?;
req_info.bucket = Some(facts.bucket.to_owned());
req_info.object = Some(facts.key.to_owned());
req_info.version_id = version;
set_requested_object_lock_retain_until(req, facts.retain_until);
authorize_request(req, Action::S3Action(S3Action::PutObjectAction)).await?;
authorize_replication_only_put_headers(req).await?;
if facts.legal_hold_present {
authorize_request(req, Action::S3Action(S3Action::PutObjectLegalHoldAction)).await?;
}
if facts.retention_present {
authorize_request(req, Action::S3Action(S3Action::PutObjectRetentionAction)).await?;
}
req.install_bucket_generation(bucket_generation?);
Ok(())
}
/// The authorization of ListObjectsV2: the object listing's, with the bucket generation an
/// on-demand migration merge binds to captured before the decision and handed on after an allow.
///
@@ -231,3 +269,7 @@ async fn prepare_table_data_plane_list_access<R: AuthorizationRequest + ?Sized>(
req.install_table_list_access(access);
Ok(())
}
#[cfg(test)]
#[path = "listing_multipart_test.rs"]
mod tests;
@@ -0,0 +1,108 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Multipart initiation policy decisions take precedence over generation-read failures.
use super::{CreateMultipartAuthorization, authorize_create_multipart_upload};
use crate::app::RequestEnvelope;
use crate::runtime_sources::{AppContext, ServerContextSlot};
use crate::storage::access::tests::{TestKms, UnreadyIam, build_request, ensure_req_info};
use crate::storage::access::{
AuthorizationRequest, BucketGenerationGuard, CreateMultipartUploadInput, FS, ObjectLockLegalHoldStatus, ObjectLockMode,
S3Access as _,
};
use crate::storage::storage_api::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use http::{HeaderValue, Method};
use std::sync::Arc;
use time::OffsetDateTime;
#[test]
#[serial_test::serial]
fn create_multipart_authorization_surfaces_generation_errors_after_policy_checks() {
crate::app::gating_test_env::run_large_stack_test("create-multipart-authorization-order", || async {
const ACTIONS: &[&str] = &[
"s3:PutObject",
"s3:ReplicateObject",
"s3:PutObjectLegalHold",
"s3:PutObjectRetention",
];
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
let slot = ServerContextSlot::new();
let fs = FS::with_server_ctx(slot.clone());
assert!(slot.install(Arc::new(AppContext::new(store.clone(), Arc::new(UnreadyIam), Arc::new(TestKms)))));
let bucket = format!("create-auth-order-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create test bucket");
for excluded in ACTIONS.iter().copied().map(Some).chain([None]) {
let policy = serde_json::json!({"Version":"2012-10-17","Statement":[{
"Effect":"Allow","Principal":{"AWS":"*"},
"Action":ACTIONS.iter().copied().filter(|action| Some(*action) != excluded).collect::<Vec<_>>(),
"Resource":format!("arn:aws:s3:::{bucket}/object")
}]});
store
.update_bucket_metadata_config(
&bucket,
crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG,
serde_json::to_vec(&policy).unwrap(),
)
.await
.expect("publish policy");
let retain_until = OffsetDateTime::now_utc() + time::Duration::days(1);
let mut legacy = build_request(
CreateMultipartUploadInput {
bucket: bucket.clone(),
key: "object".to_owned(),
object_lock_legal_hold_status: Some(ObjectLockLegalHoldStatus::from_static("OFF")),
object_lock_mode: Some(ObjectLockMode::from_static("GOVERNANCE")),
object_lock_retain_until_date: Some(retain_until.into()),
..Default::default()
},
Method::POST,
);
legacy
.headers
.insert("x-rustfs-source-replication-request", HeaderValue::from_static("true"));
legacy.extensions.insert(slot.clone());
ensure_req_info(&mut legacy);
legacy.install_bucket_generation(BucketGenerationGuard {
bucket: "other-bucket".to_owned(),
incarnation_id: uuid::Uuid::new_v4(),
});
let mut owned = RequestEnvelope::from_s3s(&legacy);
let legacy = fs
.create_multipart_upload(&mut legacy)
.await
.expect_err("policy or generation refusal");
let owned = authorize_create_multipart_upload(
&slot,
&mut owned,
CreateMultipartAuthorization {
bucket: &bucket,
key: "object",
query_version: None,
retain_until: Some(retain_until),
legal_hold_present: true,
retention_present: true,
},
)
.await
.expect_err("policy or generation refusal");
let expected = if excluded.is_some() { "AccessDenied" } else { "InternalError" };
assert_eq!(legacy.code().as_str(), expected, "legacy {excluded:?}");
assert_eq!(owned.code().as_str(), expected, "owned {excluded:?}");
}
});
}
@@ -23,10 +23,25 @@
use super::super::{
Action, ApiError, AuthorizationRequest, S3Action, S3Error, S3ErrorCode, S3Result, ServerContextSlot,
authorize_replication_only_put_headers, authorize_request, authorize_write_etag_condition, has_write_offset_bytes_header,
load_bucket_generation, missing_req_info_error, set_requested_object_lock_retain_until,
authorize_replication_only_put_headers, authorize_request, authorize_staged_multipart_part, authorize_write_etag_condition,
has_write_offset_bytes_header, load_bucket_generation, missing_req_info_error, set_requested_object_lock_retain_until,
};
pub(crate) async fn authorize_upload_part<R: AuthorizationRequest + ?Sized>(
req: &mut R,
server: &ServerContextSlot,
bucket: &str,
key: &str,
) -> S3Result<()> {
let generation = load_bucket_generation(server, req, bucket).await;
let info = req.req_info_mut().ok_or_else(missing_req_info_error)?;
info.bucket = Some(bucket.to_owned());
info.object = Some(key.to_owned());
authorize_staged_multipart_part(req).await?;
req.install_bucket_generation(generation?);
Ok(())
}
/// Request facts needed by both PutObject authorization edges.
pub(crate) struct PutObjectAuthorization<'a> {
pub(crate) bucket: &'a str,
@@ -16,8 +16,8 @@
//! AbortMultipartUpload: batch B2 of the gateway migration (rustfs/backlog#2765).
//!
//! `impl S3 for FS` in `ecfs.rs` calls the `*_edge` fn of each operation here. A migrated
//! operation converts its request into the gateway's input, runs the owned use case and converts
//! the answer back, then applies what the legacy encoder needs and the gateway's encoder does
//! operation projects the request into owned values, runs the use case and converts the answer
//! back, then applies what the legacy encoder needs and the gateway's encoder does
//! itself: `encoding-type=url` (legacy RustFS's members and per-segment rule, the one the gateway
//! applies under its RustFS profile) and XML escaping of entity tags. The batch that moves these
//! operations onto the gateway owns this file and is the only one that edits it.
@@ -25,7 +25,9 @@
use crate::storage::ecfs::*;
use crate::storage::helper::audit_only_operation;
use crate::storage::s3_api::bucket::encode_list_output_value;
use crate::storage::s3_api::{self, mark_on_demand_migration_list_local_only};
use crate::storage::s3_api::{
self, CreateMultipartUploadFields, DefaultMultipartUsecase, mark_on_demand_migration_list_local_only,
};
use rustfs_gateway_types::compat::s3s_5761ddfe::generated::ops::{
list_multipart_uploads, list_object_versions, list_objects, list_objects_v2,
};
@@ -33,6 +35,51 @@ use rustfs_io_metrics::record_s3_op;
use rustfs_s3_ops::S3Operation;
use tracing::instrument;
impl CreateMultipartUploadFields {
pub(crate) fn from_legacy(input: CreateMultipartUploadInput) -> Self {
Self {
bucket: input.bucket,
key: input.key,
metadata: input.metadata,
tagging: input.tagging,
storage_class: input.storage_class.map(|value| value.as_str().to_owned()),
server_side_encryption: input.server_side_encryption.map(|value| value.as_str().to_owned()),
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
ssekms_key_id: input.ssekms_key_id,
object_lock_legal_hold_status: input.object_lock_legal_hold_status.map(|value| value.as_str().to_owned()),
object_lock_mode: input.object_lock_mode.map(|value| value.as_str().to_owned()),
object_lock_retain_until_date: input.object_lock_retain_until_date.map(time::OffsetDateTime::from),
}
}
}
impl DefaultMultipartUsecase {
pub(crate) async fn execute_create_multipart_upload_from_s3s(
&self,
mut req: S3Request<CreateMultipartUploadInput>,
) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
let envelope = RequestEnvelope::from_s3s(&req);
let query_version = req.input.version_id.take();
let output = self
.execute_create_multipart_upload(envelope, CreateMultipartUploadFields::from_legacy(req.input), query_version)
.await?;
Ok(S3Response::new(CreateMultipartUploadOutput {
bucket: Some(output.bucket),
key: Some(output.key),
upload_id: Some(output.upload_id),
server_side_encryption: output.server_side_encryption.map(Into::into),
sse_customer_algorithm: output.sse_customer_algorithm,
sse_customer_key_md5: output.sse_customer_key_md5,
ssekms_key_id: output.ssekms_key_id,
checksum_algorithm: output.checksum_algorithm.map(Into::into),
checksum_type: output.checksum_type.map(Into::into),
..Default::default()
}))
}
}
impl FS {
#[instrument(name = "abort_multipart_upload", level = "debug", skip(self, req))]
pub(in crate::storage::ecfs) async fn abort_multipart_upload_edge(
@@ -49,8 +96,10 @@ impl FS {
req: S3Request<CreateMultipartUploadInput>,
) -> S3Result<S3Response<CreateMultipartUploadOutput>> {
crate::hp_guard!("S3::create_multipart_upload");
let usecase = s3_api::multipart_usecase_for(self);
usecase.execute_create_multipart_upload(req).await
record_s3_op(S3Operation::CreateMultipartUpload);
s3_api::multipart_usecase_for(self)
.execute_create_multipart_upload_from_s3s(req)
.await
}
pub(in crate::storage::ecfs) async fn list_multipart_uploads_edge(
+60 -1
View File
@@ -50,7 +50,10 @@ impl FS {
}
use crate::app::RequestEnvelope;
use crate::storage::s3_api::{DefaultObjectUsecase, PutObjectWriteInput, PutObjectWriteOutput};
use crate::storage::s3_api::{
DefaultMultipartUsecase, DefaultObjectUsecase, PutObjectWriteInput, PutObjectWriteOutput, UploadPartTarget,
UploadPartWriteInput, UploadPartWriteOutput,
};
impl PutObjectWriteInput {
pub(crate) fn from_legacy(input: PutObjectInput, trailing_headers: Option<TrailingHeaders>) -> Self {
@@ -137,6 +140,62 @@ impl DefaultObjectUsecase {
}
}
impl UploadPartWriteInput {
pub(crate) fn from_legacy(input: UploadPartInput, trailers: Option<TrailingHeaders>) -> Self {
Self {
target: UploadPartTarget::Legacy {
bucket: input.bucket,
key: input.key,
upload_id: input.upload_id,
},
part_number: input.part_number,
content_length: input.content_length,
body: input.body.map(s3_api::legacy_put_body),
trailing_headers: crate::app::trailer_adapter::trailer_source(trailers),
frame_trailers: None,
content_md5: input.content_md5,
checksum_algorithm: input.checksum_algorithm.map(|value| value.as_str().to_owned()),
checksum_crc32: input.checksum_crc32,
checksum_crc32c: input.checksum_crc32c,
checksum_crc64nvme: input.checksum_crc64nvme,
checksum_sha1: input.checksum_sha1,
checksum_sha256: input.checksum_sha256,
sse_customer_algorithm: input.sse_customer_algorithm,
sse_customer_key: input.sse_customer_key,
sse_customer_key_md5: input.sse_customer_key_md5,
}
}
}
fn upload_part_output_to_legacy(output: UploadPartWriteOutput) -> S3Response<UploadPartOutput> {
let mut response = S3Response::new(UploadPartOutput {
e_tag: output.e_tag.map(|tag| crate::storage::storage_api::to_s3s_etag(&tag)),
checksum_crc32: output.checksum_crc32,
checksum_crc32c: output.checksum_crc32c,
checksum_crc64nvme: output.checksum_crc64nvme,
checksum_sha1: output.checksum_sha1,
checksum_sha256: output.checksum_sha256,
server_side_encryption: output.server_side_encryption.map(ServerSideEncryption::from),
ssekms_key_id: output.ssekms_key_id,
sse_customer_algorithm: output.sse_customer_algorithm,
sse_customer_key_md5: output.sse_customer_key_md5,
..Default::default()
});
response.headers = output.headers;
response
}
impl DefaultMultipartUsecase {
pub async fn execute_upload_part(&self, req: S3Request<UploadPartInput>) -> S3Result<S3Response<UploadPartOutput>> {
let envelope = RequestEnvelope::from_s3s(&req);
let input = UploadPartWriteInput::from_legacy(req.input, req.trailing_headers);
self.execute_upload_part_owned(envelope, input)
.await
.map(upload_part_output_to_legacy)
.map_err(Into::into)
}
}
#[cfg(test)]
mod native_put_tests {
use super::{PutObjectInput, PutObjectWriteInput, PutObjectWriteOutput, put_output_to_legacy};
+3 -1
View File
@@ -19,7 +19,9 @@
use crate::app::bucket_usecase::DefaultBucketUsecase;
pub(crate) use crate::app::bucket_usecase::{PreparedLifecycleConfiguration, validate_lifecycle_rule_status};
use crate::app::multipart_usecase::DefaultMultipartUsecase;
pub(crate) use crate::app::multipart_usecase::{
CreateMultipartUploadFields, DefaultMultipartUsecase, UploadPartTarget, UploadPartWriteInput, UploadPartWriteOutput,
};
pub(crate) use crate::app::object::reject_veeam_sosapi_mutation;
pub(crate) use crate::app::object::shared::mark_on_demand_migration_list_local_only;
pub(crate) use crate::app::object::{PutObjectWriteInput, PutObjectWriteOutput, legacy_put_body};
+4 -4
View File
@@ -537,8 +537,8 @@ pub(crate) mod ecstore_config {
pub(crate) mod ecstore_data_usage {
pub(crate) use rustfs_ecstore::api::data_usage::{
apply_bucket_usage_memory_overlay, get_bucket_usage_memory, init_compression_total_memory_from_backend,
load_admin_data_usage_from_backend_cached, load_bucket_usage_from_backend_cached, load_data_usage_from_backend,
get_bucket_usage_memory, init_compression_total_memory_from_backend, load_admin_data_usage_from_backend_cached,
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_data_usage_from_backend,
lookup_degraded_bucket_usage_baseline, quota_object_size, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, store_compression_total_in_backend,
@@ -546,8 +546,8 @@ pub(crate) mod ecstore_data_usage {
// Test-only observables for the rustfs/backlog#1306 revert detector.
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::data_usage::{
compute_bucket_usage, live_bucket_usage_computations, load_data_usage_from_backend_cached,
seed_bucket_usage_memory_for_test, store_data_usage_in_backend,
apply_bucket_usage_memory_overlay, compute_bucket_usage, live_bucket_usage_computations,
load_data_usage_from_backend_cached, seed_bucket_usage_memory_for_test, store_data_usage_in_backend,
};
}
+5 -2
View File
@@ -245,8 +245,11 @@ pub(crate) mod server {
#[cfg(test)]
pub(crate) mod test_support {
pub(crate) use crate::storage::storage_api::access_consumer::{AuthorizationRequest, ReqInfo};
pub(crate) use crate::storage::storage_api::contract::bucket::{BucketOperations, MakeBucketOptions};
pub(crate) use crate::storage::storage_api::contract::object::ObjectOperations;
pub(crate) use crate::storage::storage_api::contract::bucket::{
BucketOperations, DeleteBucketOptions, MakeBucketOptions,
};
pub(crate) use crate::storage::storage_api::contract::multipart::{CompletePart, MultipartOperations};
pub(crate) use crate::storage::storage_api::contract::object::{ObjectIO, ObjectOperations};
pub(crate) use crate::storage::storage_api::{
FS, StorageObjectOptions as ObjectOptions, get_bucket_metadata, set_bucket_metadata,
};
@@ -25,7 +25,17 @@ use rustfs_ecstore::api::object::{ObjectOptions, PutObjReader};
use rustfs_s3_ops::S3Operation;
use rustfs_storage_api::{MultipartOperations as _, ObjectIO as _};
mod create;
pub(crate) const MATRIX: &[MatrixRow] = &[
MatrixRow {
operation: "CreateMultipartUpload",
metric: S3Operation::CreateMultipartUpload,
has_error_body: true,
call: |client, bucket| {
Box::pin(async move { outcome(client.create_multipart_upload().bucket(bucket).key("matrix").send().await) })
},
},
MatrixRow {
operation: "ListObjectsV2",
metric: S3Operation::ListObjectsV2,
@@ -0,0 +1,764 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
//! CreateMultipartUpload wire replies and durable session facts.
use super::super::{GATEWAY, LEGACY, create_bucket_in_the_store, s3_client, with_stack};
use aws_sdk_s3::config::interceptors::AfterDeserializationInterceptorContextRef;
use aws_sdk_s3::config::{ConfigBag, Intercept, RuntimeComponents};
use aws_sdk_s3::error::{BoxError, ProvideErrorMetadata};
use rustfs::app::context::get_global_app_context;
use rustfs::embedded::RustFSServer;
use rustfs_storage_api::MultipartOperations as _;
use std::sync::{Arc, Mutex};
const BUCKET: &str = "create-multipart-wire";
const LOCK_BUCKET: &str = "create-multipart-locked";
const RESERVED_BUCKET: &str = "sthree-multipart";
const STORAGE_CLASS: &str = "x-amz-storage-class";
const LEGAL_HOLD: &str = "x-amz-object-lock-legal-hold";
const LOCK_MODE: &str = "x-amz-object-lock-mode";
const RETAIN_UNTIL: &str = "x-amz-object-lock-retain-until-date";
const DATE: &str = "2030-01-01T00:00:00.123456789Z";
struct Case {
name: &'static str,
bucket: &'static str,
key: &'static str,
headers: &'static [(&'static str, &'static str)],
}
const CASES: &[Case] = &[
Case {
name: "plain",
bucket: BUCKET,
key: "plain",
headers: &[
(STORAGE_CLASS, "STANDARD"),
("content-type", "application/x-multipart-fixture"),
("x-amz-meta-content-type", "user-content-type"),
("x-amz-tagging", "project=rustfs"),
("x-amz-checksum-algorithm", "CRC64NVME"),
("x-amz-checksum-type", "FULL_OBJECT"),
],
},
Case {
name: "reserved",
bucket: RESERVED_BUCKET,
key: "a%252Fb%20%2B%26%3C%E9%9B%AA",
headers: &[("x-amz-acl", "unknown-acl"), ("x-amz-request-payer", "unknown-payer")],
},
Case {
name: "backslash",
bucket: BUCKET,
key: "a%5Cb",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "tab",
bucket: BUCKET,
key: "a%09%20b",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "double-slash",
bucket: BUCKET,
key: "dir//x",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "dot-dot",
bucket: BUCKET,
key: "dir/../x",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "internal-name",
bucket: BUCKET,
key: "xl.meta",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "control",
bucket: BUCKET,
key: "a%01b",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "empty",
bucket: BUCKET,
key: "",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "nul",
bucket: BUCKET,
key: "a%00b",
headers: &[(STORAGE_CLASS, "unknown-class")],
},
Case {
name: "unknown-sse",
bucket: BUCKET,
key: "sse",
headers: &[("x-amz-server-side-encryption", "unknown-sse")],
},
Case {
name: "unlocked",
bucket: BUCKET,
key: "lock",
headers: &[(LEGAL_HOLD, "unknown-hold")],
},
Case {
name: "lock-mode-only",
bucket: LOCK_BUCKET,
key: "missing-date",
headers: &[(LOCK_MODE, "unknown-mode")],
},
Case {
name: "lock-mode-with-date",
bucket: LOCK_BUCKET,
key: "invalid-mode",
headers: &[(LOCK_MODE, "unknown-mode"), (RETAIN_UNTIL, DATE)],
},
Case {
name: "hold-with-date",
bucket: LOCK_BUCKET,
key: "invalid-hold",
headers: &[(LEGAL_HOLD, "unknown-hold"), (LOCK_MODE, "GOVERNANCE"), (RETAIN_UNTIL, DATE)],
},
Case {
name: "class-before-lock",
bucket: LOCK_BUCKET,
key: "invalid-class",
headers: &[(STORAGE_CLASS, "unknown-class"), (LEGAL_HOLD, "unknown-hold")],
},
Case {
name: "utc-date",
bucket: LOCK_BUCKET,
key: "utc-date",
headers: &[(LOCK_MODE, "GOVERNANCE"), (RETAIN_UNTIL, DATE)],
},
Case {
name: "offset-date",
bucket: LOCK_BUCKET,
key: "offset-date",
headers: &[
(LOCK_MODE, "GOVERNANCE"),
(RETAIN_UNTIL, "2030-01-01T08:00:00.123456789+08:00"),
],
},
Case {
name: "invalid-checksum",
bucket: BUCKET,
key: "invalid-checksum",
headers: &[("x-amz-checksum-algorithm", "SHA256"), ("x-amz-checksum-type", "FULL_OBJECT")],
},
];
#[derive(Clone, Debug, Default)]
struct SuccessWire(Arc<Mutex<Option<(u16, String)>>>);
impl Intercept for SuccessWire {
fn name(&self) -> &'static str {
"create-multipart-response"
}
fn read_after_deserialization(
&self,
context: &AfterDeserializationInterceptorContextRef<'_>,
_runtime_components: &RuntimeComponents,
_config: &mut ConfigBag,
) -> Result<(), BoxError> {
let response = context.response();
let status = response.status().as_u16();
if (200..300).contains(&status) {
assert!(response.headers().get("x-amz-server-side-encryption-customer-key").is_none());
let xml = std::str::from_utf8(response.body().bytes().expect("buffered CreateMultipartUpload XML"))?;
*self.0.lock().expect("wire capture") = Some((status, xml.to_owned()));
}
Ok(())
}
}
fn expected_refusal(name: &str, legacy: bool) -> Option<(u16, &'static str, &'static str)> {
const STORAGE: (u16, &str, &str) = (400, "InvalidStorageClass", "The storage class you specified is not valid.");
const LOCK_XML: (u16, &str, &str) = (
400,
"MalformedXML",
"The XML you provided was not well-formed or did not validate against our published schema",
);
Some(match name {
"plain" | "reserved" | "utc-date" | "offset-date" => return None,
"backslash" | "tab" | "class-before-lock" => STORAGE,
"double-slash" | "dot-dot" => (400, "InvalidArgument", "Invalid object key"),
"internal-name" => (400, "InvalidArgument", "Object key contains a reserved storage filename"),
"control" => (400, "InvalidArgument", "Object key contains characters unsupported by XML listings"),
"empty" => (501, "NotImplemented", "Unknown operation"),
"nul" if !legacy => (400, "InvalidArgument", "the name contains a NUL byte"),
"nul" => (400, "InvalidArgument", "Object key contains invalid control characters: \"a\\0b\""),
"unknown-sse" if !legacy => (
400,
"InvalidArgument",
"x-amz-server-side-encryption must be one of AES256, aws:fsx, aws:kms or aws:kms:dsse",
),
"unknown-sse" => (
400,
"InvalidArgument",
"The SSE algorithm specified is not supported. The valid values are AES256 or aws:kms.",
),
"unlocked" => (400, "InvalidRequest", "Bucket is missing ObjectLockConfiguration"),
"lock-mode-only" => (
400,
"InvalidRequest",
"x-amz-object-lock-retain-until-date and x-amz-object-lock-mode must both be supplied",
),
"lock-mode-with-date" | "hold-with-date" => LOCK_XML,
"invalid-checksum" => (400, "InvalidArgument", "Invalid checksum type"),
other => panic!("no measured answer for {other}"),
})
}
async fn create_multipart_wire(server: RustFSServer, legacy: bool) {
let client = s3_client(&server.endpoint(), server.access_key(), server.secret_key());
for bucket in [BUCKET, RESERVED_BUCKET] {
create_bucket_in_the_store(bucket).await;
}
client
.create_bucket()
.bucket(LOCK_BUCKET)
.object_lock_enabled_for_bucket(true)
.send()
.await
.expect("create lock bucket");
let context = get_global_app_context().expect("embedded context");
let store = context.object_store();
let mut expected = Vec::new();
let mut observed = Vec::new();
for case in CASES {
if !legacy && case.name == "empty" {
// The empty path does not dispatch CreateMultipartUpload.
continue;
}
let uploads = || store.list_multipart_uploads(case.bucket, "", None, None, None, 100);
let before = uploads().await.expect("list before request").uploads.len();
let wire = SuccessWire::default();
let result = client
.create_multipart_upload()
.bucket(case.bucket)
.key("fixture")
.customize()
.interceptor(wire.clone())
.mutate_request(move |request| {
let (prefix, _) = request.uri().rsplit_once("/fixture").expect("SDK fixture path");
request
.set_uri(format!("{prefix}/{}?uploads", case.key))
.expect("fixture request target");
for (name, value) in case.headers {
request.headers_mut().insert(*name, *value);
}
})
.send()
.await;
let after = uploads().await.expect("list after request").uploads.len();
expected.push((
case.name,
expected_refusal(case.name, legacy)
.map(|(status, code, message)| (status, Some(code.to_owned()), Some(message.to_owned()))),
));
let output = match result {
Ok(output) => {
observed.push((case.name, None));
output
}
Err(error) => {
assert_eq!(after, before, "{}", case.name);
let response = error.raw_response().expect("wire response");
observed.push((
case.name,
Some((
response.status().as_u16(),
error.code().map(str::to_owned),
error.message().map(str::to_owned),
)),
));
for name in ["x-amz-request-id", "x-request-id"] {
assert!(
response.headers().get(name).is_some_and(|value| !value.is_empty()),
"{name}: {}",
case.name
);
}
continue;
}
};
let (status, xml) = wire.0.lock().expect("wire capture").take().expect("success XML");
assert_eq!(status, 200);
assert!(xml.contains("<InitiateMultipartUploadResult") && xml.contains("</InitiateMultipartUploadResult>"));
let key = output.key().expect("response key");
let upload_id = output.upload_id().filter(|id| !id.is_empty()).expect("nonempty upload ID");
assert_eq!(output.bucket(), Some(case.bucket));
assert!(xml.contains(&format!("<UploadId>{upload_id}</UploadId>")));
assert_eq!(after, before + 1, "{}", case.name);
let info = store
.get_multipart_info(case.bucket, key, upload_id, &Default::default())
.await
.expect("get durable upload metadata");
let metadata = |name| info.user_defined.get(name).map(String::as_str);
match case.name {
"plain" => {
assert_eq!(key, "plain");
assert_eq!(metadata("content-type"), Some("application/x-multipart-fixture"));
assert_eq!(metadata("x-amz-meta-content-type"), Some("user-content-type"));
assert_eq!(metadata("x-amz-tagging"), Some("project=rustfs"));
assert_eq!(metadata("x-rustfs-multipart-checksum"), Some("CRC64NVME"));
assert_eq!(metadata("x-rustfs-multipart-checksum-type"), Some("FULL_OBJECT"));
assert_eq!(output.checksum_algorithm().map(|value| value.as_str()), Some("CRC64NVME"));
assert_eq!(output.checksum_type().map(|value| value.as_str()), Some("FULL_OBJECT"));
}
"reserved" => {
assert_eq!(key, "a%2Fb +&<雪");
assert_eq!(metadata("content-type"), Some("binary/octet-stream"));
assert_eq!(output.checksum_algorithm(), None);
}
"utc-date" | "offset-date" => {
use time::{OffsetDateTime, format_description::well_known::Rfc3339};
assert_eq!(key, case.key);
assert_eq!(metadata(LOCK_MODE), Some("GOVERNANCE"));
let date = metadata(RETAIN_UNTIL).expect("durable retention date");
let requested = case.headers.iter().find(|(name, _)| *name == RETAIN_UNTIL).unwrap().1;
if legacy {
assert_eq!(date, requested);
}
let nanos = |value| OffsetDateTime::parse(value, &Rfc3339).unwrap().unix_timestamp_nanos();
assert_eq!(nanos(date), nanos(requested));
}
other if expected_refusal(other, legacy).is_some() => {}
other => panic!("no durable assertion for {other}"),
}
}
assert_eq!(observed, expected);
server.shutdown().await;
}
#[test]
fn create_multipart_baseline_legacy_wire() {
with_stack(LEGACY, |server| create_multipart_wire(server, true));
}
#[test]
fn create_multipart_wire_gateway() {
with_stack(GATEWAY, |server| create_multipart_wire(server, false));
}
async fn create_multipart_authorization(server: RustFSServer) {
use super::super::{USER_SECRET_KEY, create_user_without_a_policy};
use aws_sdk_s3::Client;
use rustfs_iam::store::UserType;
use rustfs_policy::policy::Policy;
use serde_json::json;
const PUT_ONLY: &str = "multipart-put-only";
const FULL_ACCESS: &str = "multipart-full-access";
const NO_POLICY: &str = "multipart-no-policy";
const TARGET: &str = "target";
const VERSION: &str = "0194e0f1-0000-7000-8000-000000000000";
const REFUSED_VERSION: &str = "0194e0f1-0000-7000-8000-000000000001";
const REPLICATION: &str = "x-rustfs-source-replication-request";
let endpoint = server.endpoint();
let root = s3_client(&endpoint, server.access_key(), server.secret_key());
root.create_bucket()
.bucket(LOCK_BUCKET)
.object_lock_enabled_for_bucket(true)
.send()
.await
.expect("create the Object Lock bucket");
create_bucket_in_the_store(BUCKET).await;
let context = get_global_app_context().expect("embedded context");
let store = context.object_store();
let iam = context.iam().handle();
for (user, actions) in [
(PUT_ONLY, &["s3:PutObject"][..]),
(
FULL_ACCESS,
&[
"s3:PutObject",
"s3:ReplicateObject",
"s3:PutObjectLegalHold",
"s3:PutObjectRetention",
][..],
),
] {
create_user_without_a_policy(user).await;
let resource = format!("arn:aws:s3:::{LOCK_BUCKET}/{TARGET}");
let policy = json!({"Version":"2012-10-17","Statement":[
{"Effect":"Allow","Action":actions,"Resource":resource},
{"Effect":"Deny","Action":["s3:PutObject"],"Resource":resource,
"Condition":{"StringEquals":{"s3:VersionId":REFUSED_VERSION}}}
]});
let policy = Policy::parse_config(policy.to_string().as_bytes()).expect("valid test policy");
iam.set_policy(user, policy).await.expect("store the test policy");
iam.policy_db_set(user, UserType::Reg, false, user)
.await
.expect("attach the test policy");
}
create_user_without_a_policy(NO_POLICY).await;
let put_only = s3_client(&endpoint, PUT_ONLY, USER_SECRET_KEY);
let full = s3_client(&endpoint, FULL_ACCESS, USER_SECRET_KEY);
let no_policy = s3_client(&endpoint, NO_POLICY, USER_SECRET_KEY);
struct Attempt<'a> {
reason: &'static str,
client: &'a Client,
bucket: &'static str,
key: &'static str,
version: Option<&'static str>,
headers: &'static [(&'static str, &'static str)],
refusal: Option<(&'static str, u16)>,
}
let denied = Some(("AccessDenied", 403));
for row in [
Attempt {
reason: "no PutObject permission",
client: &no_policy,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[],
refusal: denied,
},
Attempt {
reason: "wrong bucket",
client: &put_only,
bucket: BUCKET,
key: TARGET,
version: None,
headers: &[],
refusal: denied,
},
Attempt {
reason: "wrong key",
client: &put_only,
bucket: LOCK_BUCKET,
key: "other",
version: None,
headers: &[],
refusal: denied,
},
Attempt {
reason: "no ReplicateObject permission",
client: &put_only,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[(REPLICATION, "true")],
refusal: denied,
},
Attempt {
reason: "no PutObjectLegalHold permission",
client: &put_only,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[(LEGAL_HOLD, "OFF")],
refusal: denied,
},
Attempt {
reason: "no PutObjectRetention permission",
client: &put_only,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[(LOCK_MODE, "GOVERNANCE"), (RETAIN_UNTIL, DATE)],
refusal: denied,
},
Attempt {
reason: "query version overrides header on deny",
client: &full,
bucket: LOCK_BUCKET,
key: TARGET,
version: Some(REFUSED_VERSION),
headers: &[("x-rustfs-source-version-id", VERSION)],
refusal: denied,
},
Attempt {
reason: "MinIO header alias supplies the policy version",
client: &full,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[("x-minio-source-version-id", REFUSED_VERSION)],
refusal: denied,
},
Attempt {
reason: "valid scoped PutObject",
client: &put_only,
bucket: LOCK_BUCKET,
key: TARGET,
version: None,
headers: &[],
refusal: None,
},
Attempt {
reason: "replication, legal hold and retention allowed",
client: &full,
bucket: LOCK_BUCKET,
key: TARGET,
version: Some(VERSION),
headers: &[
("x-minio-source-version-id", REFUSED_VERSION),
(LEGAL_HOLD, "OFF"),
(LOCK_MODE, "GOVERNANCE"),
(RETAIN_UNTIL, DATE),
],
refusal: None,
},
Attempt {
reason: "null query version overrides the header",
client: &full,
bucket: LOCK_BUCKET,
key: TARGET,
version: Some("null"),
headers: &[("x-rustfs-source-version-id", REFUSED_VERSION)],
refusal: None,
},
Attempt {
reason: "a query version still needs replication authorization",
client: &put_only,
bucket: LOCK_BUCKET,
key: TARGET,
version: Some(VERSION),
headers: &[],
refusal: Some(("InvalidArgument", 400)),
},
] {
let uploads = || store.list_multipart_uploads(row.bucket, "", None, None, None, 100);
let before = uploads().await.expect("list before authorization").uploads.len();
let result = row
.client
.create_multipart_upload()
.bucket(row.bucket)
.key(row.key)
.customize()
.mutate_request(move |request| {
if let Some(version) = row.version {
let uri = format!("{}&versionId={version}", request.uri());
request.set_uri(uri).expect("CreateMultipartUpload with versionId");
}
for (name, value) in row.headers {
request.headers_mut().insert(*name, *value);
}
})
.send()
.await;
let after = uploads().await.expect("list after authorization").uploads.len();
if let Some((code, status)) = row.refusal {
let error = result.expect_err(row.reason);
assert_eq!(super::super::refusal(&error), (Some(code.to_owned()), Some(status)), "{}", row.reason);
assert_eq!(before, after, "{}", row.reason);
} else {
let output = result.expect(row.reason);
assert_eq!(output.bucket(), Some(row.bucket), "{}", row.reason);
assert_eq!(output.key(), Some(row.key), "{}", row.reason);
assert_eq!(before + 1, after, "{}", row.reason);
if row.version == Some(VERSION) {
use rustfs_ecstore::api::object::{ObjectOptions, PutObjReader};
use rustfs_storage_api::CompletePart;
let upload_id = output.upload_id().expect("authorized upload ID");
let options = ObjectOptions {
versioned: true,
..Default::default()
};
let mut body = PutObjReader::from_vec(b"part of the requested version".to_vec());
let part = store
.put_object_part(row.bucket, row.key, upload_id, 1, &mut body, &options)
.await
.expect("stage a part for the selected version");
let completed = store
.clone()
.complete_multipart_upload(
row.bucket,
row.key,
upload_id,
vec![CompletePart {
part_num: 1,
etag: part.etag,
..Default::default()
}],
&options,
)
.await
.expect("publish the version persisted by CreateMultipartUpload");
assert_eq!(completed.version_id.expect("versioned object").to_string(), VERSION);
}
}
}
server.shutdown().await;
}
#[test]
fn create_multipart_authorization_legacy() {
with_stack(LEGACY, create_multipart_authorization);
}
#[test]
fn create_multipart_authorization_gateway() {
with_stack(GATEWAY, create_multipart_authorization);
}
async fn create_multipart_encryption(server: RustFSServer, legacy: bool) {
use rustfs_ecstore::api::bucket::{metadata::BUCKET_ETAG_CONFIG, metadata_sys};
use rustfs_utils::http::{SUFFIX_ETAG_ALGORITHM, SUFFIX_PLAINTEXT_CHECKSUM, get_consistent_str};
let bucket = "create-multipart-encryption";
create_bucket_in_the_store(bucket).await;
metadata_sys::update(bucket, BUCKET_ETAG_CONFIG, br#"{"algorithm":"xxh3"}"#.to_vec())
.await
.expect("configure bucket XXH3 ETags");
let client = s3_client(&server.endpoint(), server.access_key(), server.secret_key());
let context = get_global_app_context().expect("embedded context");
let store = context.object_store();
let kms = context.kms().handle();
kms.reconfigure(rustfs_kms::config::KmsConfig::static_kms(
"multipart-test-key".to_owned(),
base64_simd::STANDARD.encode_to_string([1_u8; 32]),
))
.await
.expect("configure test KMS");
let create = |key: &'static str| {
client
.create_multipart_upload()
.bucket(bucket)
.key(key)
.checksum_algorithm(aws_sdk_s3::types::ChecksumAlgorithm::Crc64Nvme)
.checksum_type(aws_sdk_s3::types::ChecksumType::FullObject)
};
let count = || {
rustfs_io_metrics::s3_op_metrics_snapshot()
.into_iter()
.find(|row| row.op == rustfs_s3_ops::S3Operation::CreateMultipartUpload.as_str())
.map_or(0, |row| row.total)
};
let customer_key = base64_simd::STANDARD.encode_to_string([0_u8; 32]);
let customer_md5 = "cLyPS3KoaSFGi/joRB3OUQ==";
let ssec = |md5: &'static str| {
create("sse-c")
.sse_customer_algorithm("AES256")
.sse_customer_key(customer_key.clone())
.sse_customer_key_md5(md5)
};
let uploads = || store.list_multipart_uploads(bucket, "", None, None, None, 100);
let supplied_md5 = "1B2M2Y8AsgTpgAmY7PhCfg==";
if legacy && !cfg!(feature = "rio-v2") {
let before = count();
let output = ssec(supplied_md5).send().await.expect("initiate with the supplied digest");
let upload_id = output.upload_id().expect("upload ID for the supplied digest");
let stored = uploads().await.expect("list the initiated upload").uploads;
assert_eq!(stored.len(), 1);
assert_eq!(stored[0].upload_id, upload_id);
assert_eq!(stored[0].object, "sse-c");
assert_eq!(output.sse_customer_key_md5(), Some(supplied_md5));
assert_eq!(count(), before + 1);
store
.abort_multipart_upload(bucket, "sse-c", upload_id, &Default::default())
.await
.expect("remove the supplied-digest fixture");
} else if !legacy {
let before = count();
let response = ssec(supplied_md5).send().await;
assert!(uploads().await.expect("list after a refused digest").uploads.is_empty());
let error = response.expect_err("the gateway refuses a nonmatching customer-key digest");
assert_eq!(super::super::refusal(&error), (Some("InvalidArgument".to_owned()), Some(400)));
assert_eq!(
error.message(),
Some("the customer-provided encryption key and its MD5 must be base64 of 32 and 16 bytes and must agree")
);
assert_eq!(count(), before);
}
let before = count();
let response = ssec(customer_md5)
.server_side_encryption(aws_sdk_s3::types::ServerSideEncryption::Aes256)
.send()
.await;
assert!(
uploads()
.await
.expect("list after conflicting encryption headers")
.uploads
.is_empty()
);
let error = response.expect_err("an upload selects one encryption channel");
assert_eq!(super::super::refusal(&error), (Some("InvalidArgument".to_owned()), Some(400)));
let message = if legacy {
"The SSE-C and managed server-side encryption headers cannot be used together."
} else {
"a request may name a server-managed encryption algorithm or a customer-provided key, not both"
};
assert_eq!(error.message(), Some(message));
assert_eq!(count(), before + u64::from(legacy));
let before = count();
let customer = ssec(customer_md5)
.customize()
.interceptor(SuccessWire::default())
.send()
.await
.expect("create an SSE-C upload");
assert_eq!(count(), before + 1);
assert_eq!(customer.sse_customer_algorithm(), Some("AES256"));
assert_eq!(customer.sse_customer_key_md5(), Some(customer_md5));
let before = count();
let managed = create("sse-kms")
.server_side_encryption(aws_sdk_s3::types::ServerSideEncryption::AwsKms)
.ssekms_key_id("multipart-test-key")
.send()
.await
.expect("create an SSE-KMS upload");
assert_eq!(count(), before + 1);
assert_eq!(managed.server_side_encryption().map(|value| value.as_str()), Some("aws:kms"));
assert_eq!(managed.ssekms_key_id(), Some("multipart-test-key"));
for output in [customer, managed] {
assert_eq!(output.checksum_algorithm().map(|value| value.as_str()), Some("CRC64NVME"));
assert_eq!(output.checksum_type().map(|value| value.as_str()), Some("FULL_OBJECT"));
let info = store
.get_multipart_info(
bucket,
output.key().expect("encrypted upload key"),
output.upload_id().expect("encrypted upload ID"),
&Default::default(),
)
.await
.expect("read the durable encrypted upload");
assert_eq!(get_consistent_str(&info.user_defined, SUFFIX_PLAINTEXT_CHECKSUM), Some("true"));
assert_eq!(get_consistent_str(&info.user_defined, SUFFIX_ETAG_ALGORITHM), Some("xxh3"));
}
kms.stop().await.expect("stop test KMS");
server.shutdown().await;
}
#[test]
fn create_multipart_encryption_legacy() {
with_stack(LEGACY, |server| create_multipart_encryption(server, true));
}
#[test]
fn create_multipart_encryption_gateway() {
with_stack(GATEWAY, |server| create_multipart_encryption(server, false));
}
@@ -12,26 +12,54 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! PutObject refusal rows and legacy/gateway wire behavior. UploadPart remains unserved.
//! PutObject and UploadPart refusal rows and legacy/gateway wire behavior.
pub(crate) const MATRIX: &[super::MatrixRow] = &[super::MatrixRow {
operation: "PutObject",
metric: rustfs_s3_ops::S3Operation::PutObject,
has_error_body: true,
call: |client, bucket| {
Box::pin(async move {
super::outcome(
client
.put_object()
.bucket(bucket)
.key("put-matrix")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"matrix"))
.send()
.await,
)
})
pub(crate) const MATRIX: &[super::MatrixRow] = &[
super::MatrixRow {
operation: "PutObject",
metric: rustfs_s3_ops::S3Operation::PutObject,
has_error_body: true,
call: |client, bucket| {
Box::pin(async move {
super::outcome(
client
.put_object()
.bucket(bucket)
.key("put-matrix")
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"matrix"))
.send()
.await,
)
})
},
},
}];
super::MatrixRow {
operation: "UploadPart",
metric: rustfs_s3_ops::S3Operation::UploadPart,
has_error_body: true,
call: |client, bucket| {
Box::pin(async move {
use rustfs_storage_api::MultipartOperations as _;
let store = super::get_global_app_context().expect("embedded context").object_store();
let upload = store
.new_multipart_upload(bucket, "part-matrix", &Default::default())
.await
.expect("matrix upload");
super::outcome(
client
.upload_part()
.bucket(bucket)
.key("part-matrix")
.upload_id(upload.upload_id)
.part_number(1)
.body(aws_sdk_s3::primitives::ByteStream::from_static(b"matrix"))
.send()
.await,
)
})
},
},
];
use hmac::{Hmac, KeyInit, Mac};
use sha2::{Digest, Sha256};
@@ -632,3 +660,6 @@ impl WireClient<'_> {
.await
}
}
#[path = "upload_part.rs"]
mod upload_part;
@@ -0,0 +1,275 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Storage-backed UploadPart wire tests.
use super::{CRC32_TRAILER_HEADERS, VALID_CRC32_TRAILER, WireAnswer, WireClient};
use rustfs_storage_api::MultipartOperations;
const BUCKET: &str = "upload-part-contract";
const OTHER: &str = "upload-part-other";
const KEY: &str = "part-object";
const ETAG: &str = "5d41402abc4b2a76b9719d911017c592";
const BOTH_CHECKSUMS: &[(&str, &str)] = &[
("x-amz-checksum-crc32", "NhCmhg=="),
("x-amz-checksum-sha256", "LPJNul+wow4m6DsqxbninhsWHlwfp0JecwQzYpOLmCQ="),
];
async fn upload(
wire: &WireClient<'_>,
path: &str,
id: &str,
number: i32,
body: &[u8],
headers: &[(&str, &str)],
length: Option<usize>,
) -> WireAnswer {
wire.request("PUT", &format!("/{path}?partNumber={number}&uploadId={id}"), body, headers, length)
.await
}
fn xml_field<'a>(answer: &'a WireAnswer, name: &str) -> Option<&'a str> {
let (_, tail) = answer.body.split_once(&format!("<{name}>"))?;
let (value, _) = tail.split_once(&format!("</{name}>"))?;
Some(value)
}
const MISSING_SESSION: &str = "The specified multipart upload does not exist. The upload ID may be invalid, or the upload may have been aborted or completed.";
fn assert_reply(reply: &WireAnswer, status: u16, code: Option<&str>, message: Option<&str>) {
assert_eq!((reply.status, xml_field(reply, "Code")), (status, code), "{}", reply.body);
if let Some(message) = message {
assert_eq!(xml_field(reply, "Message"), Some(message), "{}", reply.body);
}
}
fn assert_upload_success(reply: &WireAnswer, sha256: Option<&str>) {
assert_reply(reply, 200, None, None);
assert_eq!(
reply.headers.get("etag").map(String::as_str),
Some("\"5d41402abc4b2a76b9719d911017c592\"")
);
assert_eq!(reply.headers.get("x-amz-checksum-crc32").map(String::as_str), Some("NhCmhg=="));
assert_eq!(reply.headers.get("x-amz-checksum-sha256").map(String::as_str), sha256);
}
async fn upload_part_contract(server: super::super::RustFSServer, gateway: bool) {
for bucket in [BUCKET, OTHER] {
super::super::create_bucket_in_the_store(bucket).await;
}
let endpoint = server.endpoint();
let wire = WireClient {
endpoint: &endpoint,
access_key: server.access_key(),
secret_key: server.secret_key(),
};
let store = super::super::get_global_app_context().unwrap().object_store();
let options = Default::default();
let path = format!("{BUCKET}/{KEY}");
let id = store.new_multipart_upload(BUCKET, KEY, &options).await.unwrap().upload_id;
let other_id = store.new_multipart_upload(OTHER, KEY, &options).await.unwrap().upload_id;
let denied_key_id = store
.new_multipart_upload(BUCKET, "other-key", &options)
.await
.unwrap()
.upload_id;
let aborted = store.new_multipart_upload(BUCKET, KEY, &options).await.unwrap().upload_id;
store.abort_multipart_upload(BUCKET, KEY, &aborted, &options).await.unwrap();
let both = upload(&wire, &path, &id, 1, b"hello", BOTH_CHECKSUMS, Some(5)).await;
let shipped = if gateway {
assert_reply(&both, 400, Some("InvalidRequest"), None);
assert!(
store
.list_object_parts(BUCKET, KEY, &id, None, 20, &options)
.await
.unwrap()
.parts
.is_empty()
);
upload(&wire, &path, &id, 1, b"hello", &BOTH_CHECKSUMS[..1], Some(5)).await
} else {
both
};
assert_upload_success(&shipped, (!gateway).then_some(BOTH_CHECKSUMS[1].1));
let max = upload(&wire, &path, &id, 10000, VALID_CRC32_TRAILER, &CRC32_TRAILER_HEADERS, None).await;
assert_upload_success(&max, None);
let unknown_id = store
.new_multipart_upload(BUCKET, "another-key", &options)
.await
.unwrap()
.upload_id;
let wrong_key = format!("{BUCKET}/other-key");
let wrong_bucket = format!("{OTHER}/{KEY}");
let nul_key = format!("{BUCKET}/bad%00key");
let bad_checksum = &[("x-amz-checksum-crc32", "AAAAAA==")];
let wrong_width = &[("x-amz-checksum-crc32", "AAAA")];
let bad_trailer = b"5\r\nhello\r\n0\r\nx-amz-checksum-crc32:AAAAAA==\r\n\r\n";
let missing_trailer = b"5\r\nhello\r\n0\r\n\r\n";
let mut replies = Vec::new();
for (label, path, id, number, body, headers, length) in [
("zero", path.as_str(), id.as_str(), 0, b"hello".as_slice(), &[][..], Some(5)),
("negative", &path, &id, -1, b"hello", &[], Some(5)),
("above-max", &path, &id, 10001, b"hello", &[], Some(5)),
("malformed", &path, "malformed", 2, b"hello", &[], Some(5)),
("not-here", &path, &unknown_id, 2, b"hello", &[], Some(5)),
("aborted", &path, &aborted, 2, b"hello", &[], Some(5)),
("wrong-key", &wrong_key, &id, 2, b"hello", &[], Some(5)),
("wrong-bucket", &wrong_bucket, &id, 2, b"hello", &[], Some(5)),
("nul-key", &nul_key, &id, 2, b"hello", &[], Some(5)),
("unknown-length", &path, &id, 2, b"hello", &[], None),
("short", &path, &id, 2, b"abc", &[], Some(5)),
("bad-checksum", &path, &id, 2, b"hello", bad_checksum, Some(5)),
("wrong-width", &path, &id, 2, b"hello", wrong_width, Some(5)),
("bad-trailer", &path, &id, 2, bad_trailer, &CRC32_TRAILER_HEADERS, None),
("missing-trailer", &path, &id, 2, missing_trailer, &CRC32_TRAILER_HEADERS, None),
("bad-replacement", &path, &id, 1, b"world", bad_checksum, Some(5)),
] {
replies.push((label, upload(&wire, path, id, number, body, headers, length).await));
}
let responses: Vec<_> = replies
.iter()
.map(|(label, reply)| (*label, reply.status, xml_field(reply, "Code")))
.collect();
for (label, reply) in &replies {
let (status, code, message) = match *label {
"zero" | "negative" | "above-max" => (400, "InvalidArgument", "partNumber must be between 1 and 10000"),
"malformed" => (
404,
"NoSuchUpload",
"The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed.",
),
"not-here" | "aborted" | "wrong-key" | "wrong-bucket" => (404, "NoSuchUpload", MISSING_SESSION),
"nul-key" => (400, "InvalidArgument", "Invalid argument"),
"unknown-length" => (411, "MissingContentLength", "missing header: content-length"),
"short" => (
400,
"IncompleteBody",
"You did not provide the number of bytes specified by the Content-Length HTTP header.",
),
"bad-checksum" | "wrong-width" | "bad-trailer" | "missing-trailer" | "bad-replacement" => {
(400, "BadDigest", "The Content-Md5 you specified did not match what we received.")
}
_ => panic!("unexpected case {label}"),
};
assert_eq!((reply.status, xml_field(reply, "Code")), (status, Some(code)), "{label}: {responses:?}");
if !gateway {
assert_eq!(xml_field(reply, "Message"), Some(message), "{label}");
}
for header in ["etag", "x-amz-checksum-crc32", "x-amz-checksum-sha256"] {
assert!(!reply.headers.contains_key(header), "{label}: {header}");
}
}
let parts = store
.list_object_parts(BUCKET, KEY, &id, None, 20, &options)
.await
.unwrap()
.parts;
let summary: Vec<_> = parts
.iter()
.map(|part| (part.part_num, part.size, part.etag.as_deref()))
.collect();
for (bucket, key, session) in [(OTHER, KEY, &other_id), (BUCKET, "other-key", &denied_key_id)] {
assert!(
store
.list_object_parts(bucket, key, session, None, 20, &options)
.await
.unwrap()
.parts
.is_empty()
);
}
assert_eq!(summary, [(1, 5, Some(ETAG)), (10000, 5, Some(ETAG))]);
let sdk = super::super::s3_client(&endpoint, server.access_key(), server.secret_key());
let part = aws_sdk_s3::types::CompletedPart::builder().part_number(1).e_tag(ETAG).build();
sdk.complete_multipart_upload()
.bucket(BUCKET)
.key(KEY)
.upload_id(&id)
.multipart_upload(aws_sdk_s3::types::CompletedMultipartUpload::builder().parts(part).build())
.send()
.await
.unwrap();
let completed = upload(&wire, &path, &id, 2, b"hello", &[], Some(5)).await;
assert_reply(&completed, 404, Some("NoSuchUpload"), (!gateway).then_some(MISSING_SESSION));
let readback = sdk.get_object().bucket(BUCKET).key(KEY).send().await.unwrap();
assert_eq!(readback.body.collect().await.unwrap().into_bytes().as_ref(), b"hello");
server.shutdown().await;
}
#[test]
#[serial_test::serial]
fn legacy_upload_part_fixed_main_contract() {
super::super::with_stack(super::super::LEGACY, |server| upload_part_contract(server, false));
}
#[test]
#[serial_test::serial]
fn gateway_upload_part_wire_contract() {
super::super::with_stack(super::super::GATEWAY, |server| upload_part_contract(server, true));
}
async fn scoped_user(server: super::super::RustFSServer) {
const USER: &str = "uploadpartscoped";
let endpoint = server.endpoint();
for bucket in [BUCKET, OTHER] {
super::super::create_bucket_in_the_store(bucket).await;
}
super::super::create_user_allowed_on_one_bucket(USER, "s3:PutObject", &format!("{BUCKET}/{KEY}")).await;
let wire = WireClient {
endpoint: &endpoint,
access_key: USER,
secret_key: super::super::USER_SECRET_KEY,
};
let store = super::super::get_global_app_context().unwrap().object_store();
let opts = Default::default();
for (bucket, key, allowed) in [(BUCKET, KEY, true), (BUCKET, "denied", false), (OTHER, KEY, false)] {
let session = store.new_multipart_upload(bucket, key, &opts).await.unwrap().upload_id;
let answer = upload(&wire, &format!("{bucket}/{key}"), &session, 1, b"hello", &[], Some(5)).await;
let parts = store
.list_object_parts(bucket, key, &session, None, 20, &opts)
.await
.unwrap()
.parts;
let facts: Vec<_> = parts
.iter()
.map(|part| (part.part_num, part.size, part.etag.as_deref()))
.collect();
let expected = [(1, 5, Some(ETAG))];
assert_eq!(
facts.as_slice(),
if allowed { expected.as_slice() } else { &[] },
"{bucket}/{key} after HTTP {}",
answer.status,
);
if allowed {
assert_reply(&answer, 200, None, None);
} else {
assert_reply(&answer, 403, Some("AccessDenied"), None);
}
}
server.shutdown().await;
}
#[test]
#[serial_test::serial]
fn legacy_upload_part_scoped_policy() {
super::super::with_stack(super::super::LEGACY, scoped_user);
}
#[test]
#[serial_test::serial]
fn gateway_upload_part_scoped_policy() {
super::super::with_stack(super::super::GATEWAY, scoped_user);
}
+2 -1
View File
@@ -130,9 +130,10 @@ their issue closes.
| `hotpath_warp_ab_gate.sh` | dev-tool | Relative-budget gate evaluated over the warp A/B results | used by `run_hotpath_warp_ab.sh`; hotpath runbook |
| `run_internode_grpc_ab_bench.sh` | dev-tool | One-click A/B driver for the internode gRPC optimization stages | `docs/operations/internode-grpc-benchmark-runbook.md` |
| `run_internode_transport_baseline.sh` | dev-tool | Internode transport baseline runner | internode runbook; `crates/io-metrics/README.md` |
| `run_four_node_cluster_failover_bench.sh` | dev-tool | Four-node cluster failover benchmark with optional server-only image build | `docker/compose/README.md`; internode runbook |
| `run_four_node_cluster_failover_bench.sh` | dev-tool | Four-node cluster failover benchmark with configurable host ports and optional server-only image build | `docker/compose/README.md`; internode runbook |
| `run_object_batch_bench.sh` | dev-tool | Batch object benchmark runner with PUT-specific Warp summary extraction | internode + scanner runbooks |
| `test_object_batch_bench.sh` | dev-tool | Regression test for Warp PUT summary extraction | `scripts/test_object_batch_bench.sh` |
| `wait_storage_quiescence.py` | dev-tool | Fail-closed dirty/writeback and PSI stability gate before storage benchmarks | `python3 -m unittest scripts/test_wait_storage_quiescence.py` |
| `run_object_batch_bench_enhanced.sh` | dev-tool | Enhanced batch benchmark runner; hub used by the smoke rigs | hotpath runbook |
| `run_pinned_paired_abba_bench.sh` | dev-tool | Pinned RustFS/MinIO paired ABBA benchmark orchestrator for backlog#1432 | `test_pinned_paired_abba_bench.sh` |
| `prepare_conditional_read_baseline.py` | dev-tool | Installs only the matching conditional-read benchmark harness in a clean detached baseline | `docs/testing/conditional-read-benchmark.md` |
@@ -1,328 +0,0 @@
#!/usr/bin/env python3
"""Validate measured mixed-version evidence for the internode msgpack peer gate."""
from __future__ import annotations
import argparse
import hashlib
import json
import math
from pathlib import Path
import re
import sys
from typing import Any
PHASES = ("mixed-old-peer", "all-new", "rollback-old-peer")
TOPOLOGIES = {
"mixed-old-peer": (3, 1),
"all-new": (4, 0),
"rollback-old-peer": (3, 1),
}
OLD_NODES = {
"mixed-old-peer": {"node4"},
"all-new": set(),
"rollback-old-peer": {"node2"},
}
ARTIFACTS = ("decisions", "fallback", "decode_errors", "rpc_errors")
DECISIONS = {"msgpack_only", "dual_write"}
REASONS = {"manual_fleet", "peer_capable", "peer_unknown", "peer_revoked", "not_requested"}
REVISION = re.compile(r"^[0-9a-f]{40}$")
SHA256 = re.compile(r"^[0-9a-f]{64}$")
MAX_DOCUMENT_BYTES = 1024 * 1024
class EvidenceError(ValueError):
pass
def require(condition: bool, message: str) -> None:
if not condition:
raise EvidenceError(message)
def unique_object(pairs: list[tuple[str, Any]]) -> dict[str, Any]:
result: dict[str, Any] = {}
for key, value in pairs:
require(key not in result, f"duplicate JSON key: {key}")
result[key] = value
return result
def load_json(path: Path) -> tuple[dict[str, Any], str]:
require(path.is_file(), f"missing evidence artifact: {path}")
size = path.stat().st_size
require(0 < size <= MAX_DOCUMENT_BYTES, f"invalid evidence artifact size: {path}")
raw = path.read_bytes()
document = json.loads(raw, object_pairs_hook=unique_object)
require(isinstance(document, dict), f"evidence artifact must be an object: {path}")
return document, hashlib.sha256(raw).hexdigest()
def load_raw(path: Path) -> str:
require(path.is_file(), f"missing evidence artifact: {path}")
size = path.stat().st_size
require(0 < size <= MAX_DOCUMENT_BYTES, f"invalid evidence artifact size: {path}")
return hashlib.sha256(path.read_bytes()).hexdigest()
def bounded_number(value: Any, name: str) -> float:
require(type(value) in (int, float), f"{name} must be numeric")
result = float(value)
require(math.isfinite(result) and result >= 0, f"{name} must be finite and non-negative")
return result
def uint(value: Any, name: str) -> int:
require(type(value) is int and value >= 0, f"{name} must be a non-negative integer")
return value
def timestamped_value(row: dict[str, Any], start: float, end: float, lag: float, name: str) -> float:
value = row.get("value")
require(isinstance(value, list) and len(value) == 2, f"{name} must contain one Prometheus value")
timestamp = bounded_number(value[0], f"{name} timestamp")
require(end <= timestamp <= end + lag, f"{name} timestamp is outside its phase completion window")
try:
number = float(value[1])
except (TypeError, ValueError) as error:
raise EvidenceError(f"{name} value is not numeric") from error
require(math.isfinite(number) and number >= 0, f"{name} value must be finite and non-negative")
return number
def prometheus_rows(document: dict[str, Any], name: str) -> list[dict[str, Any]]:
require(document.get("status") == "success", f"{name} Prometheus query failed")
data = document.get("data")
require(isinstance(data, dict) and data.get("resultType") == "vector", f"{name} must be a Prometheus vector")
rows = data.get("result")
require(isinstance(rows, list) and len(rows) <= 64, f"{name} has invalid vector cardinality")
require(all(isinstance(row, dict) for row in rows), f"{name} contains a non-object result")
return rows
def decision_values(document: dict[str, Any], start: float, end: float, lag: float) -> dict[tuple[str, str], float]:
result: dict[tuple[str, str], float] = {}
for index, row in enumerate(prometheus_rows(document, "decisions")):
labels = row.get("metric")
require(isinstance(labels, dict) and set(labels) == {"decision", "reason"},
"decision evidence must contain only decision/reason labels")
decision = labels.get("decision")
reason = labels.get("reason")
require(decision in DECISIONS and reason in REASONS, "decision evidence contains an unbounded label")
key = (decision, reason)
require(key not in result, f"duplicate decision series: {key}")
result[key] = timestamped_value(row, start, end, lag, f"decision[{index}]")
return result
def scalar_sum(document: dict[str, Any], start: float, end: float, lag: float, name: str) -> float:
total = 0.0
rows = prometheus_rows(document, name)
require(len(rows) == 1, f"{name} scalar query must return exactly one row")
for index, row in enumerate(rows):
labels = row.get("metric")
require(labels == {}, f"{name} scalar query must not retain labels")
total += timestamped_value(row, start, end, lag, f"{name}[{index}]")
return total
def artifact_path(root: Path, relative: Any, used: set[Path], name: str) -> Path:
require(isinstance(relative, str) and relative and not Path(relative).is_absolute(), f"invalid {name} artifact path")
path = (root / relative).resolve()
try:
path.relative_to(root)
except ValueError as error:
raise EvidenceError(f"{name} artifact escapes the evidence directory") from error
require(path not in used, f"artifact reused across phases: {relative}")
used.add(path)
return path
def bool_value(value: Any, name: str) -> bool:
require(type(value) is bool, f"{name} must be boolean")
return value
def exact_env(document: Any, name: str, old: bool) -> None:
require(isinstance(document, dict), f"{name} environment must be an object")
expected = {
"msgpack_only": not old,
"peer_capability": not old,
"fleet_confirmed": False,
}
require(document == expected, f"{name} environment does not match the peer-gate canary contract")
def expected_queries(window: int) -> dict[str, str]:
return {
"decisions": (
"sum by (decision, reason) "
f"(increase(rustfs_system_network_internode_msgpack_send_decisions_total[{window}s]))"
),
"fallback": f"sum(increase(rustfs_system_network_internode_msgpack_json_fallback_total[{window}s])) or vector(0)",
"decode_errors": (
f"sum(increase(rustfs_system_network_internode_msgpack_json_decode_error_total[{window}s])) or vector(0)"
),
"rpc_errors": (
f'(sum(increase(rustfs_system_network_internode_operation_errors_total{{backend="grpc",'
f'operation!="grpc_read_version"}}[{window}s])) or vector(0)) + '
"(sum(increase(rustfs_system_network_internode_operation_classified_errors_total"
f'{{backend="grpc",operation="grpc_read_version",classification!="remote_response"}}[{window}s])) '
"or vector(0))"
),
}
def check(manifest: dict[str, Any], root: Path) -> dict[str, Any]:
root = root.resolve()
require(type(manifest.get("schema")) is int and manifest.get("schema") == 1, "unsupported canary evidence schema")
require(manifest.get("scope") == "internode-msgpack-peer-canary", "unexpected evidence scope")
source = manifest.get("source_revision")
baseline = manifest.get("baseline_revision")
require(isinstance(source, str) and REVISION.fullmatch(source) is not None, "invalid source revision")
require(isinstance(baseline, str) and REVISION.fullmatch(baseline) is not None, "invalid baseline revision")
require(source != baseline, "candidate and baseline revisions must differ")
require(bool_value(manifest.get("stable_connection_affinity"), "stable_connection_affinity"),
"stable connection-to-peer affinity is required")
lag = bounded_number(manifest.get("max_metric_lag_secs"), "max_metric_lag_secs")
require(0 <= lag <= 300, "max_metric_lag_secs exceeds the bounded limit")
phases = manifest.get("phases")
require(isinstance(phases, list) and len(phases) == len(PHASES), "exactly three canary phases are required")
used: set[Path] = set()
receipts: list[dict[str, Any]] = []
summaries: list[dict[str, Any]] = []
violations: list[str] = []
previous_end = 0.0
for expected_name, phase in zip(PHASES, phases):
require(isinstance(phase, dict) and phase.get("name") == expected_name, "canary phases are missing or out of order")
start = uint(phase.get("start_unix"), f"{expected_name} start")
end = uint(phase.get("end_unix"), f"{expected_name} end")
require(previous_end < start < end and end - start <= 3600, f"invalid {expected_name} measurement window")
previous_end = end
topology = phase.get("topology")
require(isinstance(topology, dict), f"{expected_name} topology must be an object")
expected_new, expected_old = TOPOLOGIES[expected_name]
nodes = topology.get("nodes")
require(isinstance(nodes, dict) and set(nodes) == {"node1", "node2", "node3", "node4"},
f"{expected_name} node inventory is incomplete")
expected_nodes = {
node: baseline if node in OLD_NODES[expected_name] else source
for node in ("node1", "node2", "node3", "node4")
}
require(nodes == expected_nodes, f"{expected_name} node revisions do not match the mixed-version matrix")
new_nodes = uint(topology.get("new_nodes"), f"{expected_name} new_nodes")
old_nodes = uint(topology.get("old_nodes"), f"{expected_name} old_nodes")
require(new_nodes == expected_new and old_nodes == expected_old,
f"{expected_name} topology counts do not match the four-node matrix")
environments = phase.get("environment")
require(isinstance(environments, dict), f"{expected_name} environment is missing")
exact_env(environments.get("new_nodes"), f"{expected_name} new_nodes", old=False)
exact_env(environments.get("old_nodes"), f"{expected_name} old_nodes", old=True)
queries = phase.get("queries")
require(queries == expected_queries(end - start), f"{expected_name} PromQL does not match its exact window")
workload_path = artifact_path(root, phase.get("workload"), used, f"{expected_name}.workload")
workload, workload_digest = load_json(workload_path)
summary_path = artifact_path(root, workload.get("summary"), used, f"{expected_name}.workload-summary")
summary_digest = load_raw(summary_path)
require(workload.get("summary_sha256") == summary_digest, f"{expected_name} workload summary hash mismatch")
workload_success = workload.get("status") == "success"
requests = uint(workload.get("requests"), f"{expected_name} workload requests")
require(requests > 0, f"{expected_name} workload has no requests")
if not workload_success:
violations.append(f"{expected_name}: workload failed")
rollback_readback = bool_value(workload.get("rollback_readback"), f"{expected_name} rollback_readback")
if expected_name != "rollback-old-peer":
require(not rollback_readback, f"{expected_name} cannot claim rollback readback")
if expected_name == "rollback-old-peer" and not rollback_readback:
violations.append(f"{expected_name}: rollback readback failed")
source_digest = workload.get("readback_source_sha256")
downloaded_digest = workload.get("readback_download_sha256")
if expected_name == "rollback-old-peer":
require(isinstance(source_digest, str) and SHA256.fullmatch(source_digest) is not None,
"rollback source digest is invalid")
require(source_digest == downloaded_digest, "rollback readback digest mismatch")
else:
require(source_digest is None and downloaded_digest is None,
f"{expected_name} cannot claim rollback content digests")
receipts.append({"phase": expected_name, "kind": "workload", "path": phase["workload"],
"sha256": workload_digest})
receipts.append({"phase": expected_name, "kind": "workload-summary", "path": workload["summary"],
"sha256": summary_digest})
artifacts = phase.get("artifacts")
require(isinstance(artifacts, dict) and set(artifacts) == set(ARTIFACTS), f"{expected_name} artifact set is incomplete")
documents: dict[str, dict[str, Any]] = {}
for artifact in ARTIFACTS:
path = artifact_path(root, artifacts[artifact], used, f"{expected_name}.{artifact}")
document, digest = load_json(path)
documents[artifact] = document
receipts.append({"phase": expected_name, "kind": artifact, "path": artifacts[artifact], "sha256": digest})
decisions = decision_values(documents["decisions"], start, end, lag)
peer_capable = decisions.get(("msgpack_only", "peer_capable"), 0.0)
compatibility = decisions.get(("dual_write", "peer_unknown"), 0.0) + decisions.get(
("dual_write", "peer_revoked"), 0.0
)
if peer_capable <= 0:
violations.append(f"{expected_name}: no peer_capable msgpack-only decision")
if expected_old > 0 and compatibility <= 0:
violations.append(f"{expected_name}: no old/unknown peer dual-write decision")
if decisions.get(("msgpack_only", "manual_fleet"), 0.0) > 0:
violations.append(f"{expected_name}: manual fleet path was unexpectedly active")
if decisions.get(("dual_write", "not_requested"), 0.0) > 0:
violations.append(f"{expected_name}: peer gate was not active on an eligible new-node request")
errors = {
"fallback": scalar_sum(documents["fallback"], start, end, lag, "fallback"),
"decode_errors": scalar_sum(documents["decode_errors"], start, end, lag, "decode_errors"),
"rpc_errors": scalar_sum(documents["rpc_errors"], start, end, lag, "rpc_errors"),
}
for name, value in errors.items():
if value != 0:
violations.append(f"{expected_name}: nonzero {name} ({value})")
summaries.append({
"name": expected_name,
"start_unix": start,
"end_unix": end,
"topology": topology,
"workload_requests": requests,
"peer_capable_decisions": peer_capable,
"compatibility_decisions": compatibility,
**errors,
})
return {
"observed_pass": not violations,
"scope": manifest["scope"],
"source_revision": source,
"baseline_revision": baseline,
"phases": summaries,
"violations": violations,
"artifacts": receipts,
"limits": "Measured four-node compatibility evidence only; production soak and performance drift gates remain required.",
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--manifest", type=Path, required=True)
parser.add_argument("--out", type=Path, required=True)
args = parser.parse_args()
try:
root = args.manifest.resolve().parent
manifest, digest = load_json(args.manifest.resolve())
result = check(manifest, root)
inputs = {args.manifest.resolve()} | {(root / item["path"]).resolve() for item in result["artifacts"]}
require(args.out.resolve() not in inputs, "output would overwrite raw canary evidence")
result["manifest_sha256"] = digest
args.out.write_text(json.dumps(result, indent=2, sort_keys=True, allow_nan=False) + "\n", encoding="utf-8")
return 0 if result["observed_pass"] else 1
except (EvidenceError, OSError, TypeError, KeyError, AttributeError, UnicodeError, json.JSONDecodeError) as error:
print(f"Invalid internode msgpack peer canary evidence: {error}", file=sys.stderr)
return 2
if __name__ == "__main__":
sys.exit(main())
@@ -21,6 +21,10 @@ RUSTFS_DOCKER_PLATFORM="${RUSTFS_DOCKER_PLATFORM:-}"
RUSTFS_RC_RELEASE="${RUSTFS_RC_RELEASE:-latest}"
RUSTFS_OBS_ENDPOINT="${RUSTFS_OBS_ENDPOINT:-}"
RUSTFS_UNSAFE_BYPASS_DISK_CHECK="${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
RUSTFS_NODE1_PORT="${RUSTFS_NODE1_PORT:-9000}"
RUSTFS_NODE2_PORT="${RUSTFS_NODE2_PORT:-9001}"
RUSTFS_NODE3_PORT="${RUSTFS_NODE3_PORT:-9002}"
RUSTFS_NODE4_PORT="${RUSTFS_NODE4_PORT:-9003}"
WAIT_TIMEOUT_SECS="${WAIT_TIMEOUT_SECS:-180}"
BENCH_READY_TIMEOUT_SECS="${BENCH_READY_TIMEOUT_SECS:-180}"
@@ -30,8 +34,8 @@ FAILOVER_SAMPLE_SECS="${FAILOVER_SAMPLE_SECS:-60}"
FAILOVER_INTERVAL_SECS="${FAILOVER_INTERVAL_SECS:-1}"
BENCH_WAIT_MODE="${BENCH_WAIT_MODE:-ready}"
BENCH_ENDPOINT="${BENCH_ENDPOINT:-http://127.0.0.1:9000}"
BENCH_WARP_HOSTS="${BENCH_WARP_HOSTS:-http://127.0.0.1:9000,http://127.0.0.1:9001,http://127.0.0.1:9002,http://127.0.0.1:9003}"
BENCH_ENDPOINT="${BENCH_ENDPOINT:-http://127.0.0.1:${RUSTFS_NODE1_PORT}}"
BENCH_WARP_HOSTS="${BENCH_WARP_HOSTS:-http://127.0.0.1:${RUSTFS_NODE1_PORT},http://127.0.0.1:${RUSTFS_NODE2_PORT},http://127.0.0.1:${RUSTFS_NODE3_PORT},http://127.0.0.1:${RUSTFS_NODE4_PORT}}"
BENCH_BUCKET="${BENCH_BUCKET:-rustfs-four-node-bench}"
BENCH_AUTO_NEW_BUCKET="${BENCH_AUTO_NEW_BUCKET:-true}"
BENCH_BUCKET_PREFIX="${BENCH_BUCKET_PREFIX:-rustfs-four-node-bench}"
@@ -76,6 +80,7 @@ Environment:
WITH_OBSERVABILITY BUILD_LOCAL_IMAGE RUN_FAILOVER RUN_BENCHMARK KEEP_UP
COMPOSE_UP_NO_BUILD (true|false, default: false)
RUSTFS_ACCESS_KEY RUSTFS_SECRET_KEY RUSTFS_OBS_ENDPOINT
RUSTFS_NODE1_PORT RUSTFS_NODE2_PORT RUSTFS_NODE3_PORT RUSTFS_NODE4_PORT
RUSTFS_RC_RELEASE (default: latest; disabled creates a server-only validation image)
PRECHECK_AUTO_CLEANUP (true|false, default: true)
WAIT_PROBE_MODE (service|ready, default: service)
@@ -265,7 +270,7 @@ cleanup_existing_project_containers() {
check_required_ports_free() {
local required_ports=(
9000 9001 9002 9003
"${RUSTFS_NODE1_PORT}" "${RUSTFS_NODE2_PORT}" "${RUSTFS_NODE3_PORT}" "${RUSTFS_NODE4_PORT}"
)
local occupied_ports=()
local port
@@ -345,10 +350,10 @@ run_precheck_after_build() {
node_port() {
case "$1" in
node1) echo "9000" ;;
node2) echo "9001" ;;
node3) echo "9002" ;;
node4) echo "9003" ;;
node1) echo "${RUSTFS_NODE1_PORT}" ;;
node2) echo "${RUSTFS_NODE2_PORT}" ;;
node3) echo "${RUSTFS_NODE3_PORT}" ;;
node4) echo "${RUSTFS_NODE4_PORT}" ;;
*)
log_error "unknown node name: $1 (expected node1..node4)"
exit 1
@@ -465,7 +470,7 @@ wait_node_probe_ok() {
wait_cluster_ready() {
local port
for port in 9000 9001 9002 9003; do
for port in "${RUSTFS_NODE1_PORT}" "${RUSTFS_NODE2_PORT}" "${RUSTFS_NODE3_PORT}" "${RUSTFS_NODE4_PORT}"; do
wait_node_probe_ok "${port}"
done
}
@@ -473,7 +478,7 @@ wait_cluster_ready() {
probe_survivors_ready() {
local failover_port="$1"
local port
for port in 9000 9001 9002 9003; do
for port in "${RUSTFS_NODE1_PORT}" "${RUSTFS_NODE2_PORT}" "${RUSTFS_NODE3_PORT}" "${RUSTFS_NODE4_PORT}"; do
if [[ "${port}" == "${failover_port}" ]]; then
continue
fi
@@ -761,6 +766,7 @@ main() {
export RUSTFS_SECRET_KEY
export RUSTFS_OBS_ENDPOINT
export RUSTFS_UNSAFE_BYPASS_DISK_CHECK
export RUSTFS_NODE1_PORT RUSTFS_NODE2_PORT RUSTFS_NODE3_PORT RUSTFS_NODE4_PORT
run_precheck_before_build

Some files were not shown because too many files have changed in this diff Show More