Compare commits

..

27 Commits

Author SHA1 Message Date
Chris An 4a1ce44a37 Merge remote-tracking branch 'origin/main' into fix/s3-post-object-location 2026-11-19 17:51:17 +08:00
overtrue 55be4e3f79 fix(notify): route rule engine storage imports through the storage_api boundary 2026-11-19 17:34:58 +08:00
overtrue 5be4ef65e8 test(table-catalog): issue ordinary temporary credentials through the parent-bound path
Regular parents now fence STS sessions by durable generation, so a session
stored without one is rejected as an unknown key. Publish the test sessions
with set_temp_user_from_parent, as the STS handler does.
2026-11-19 16:59:19 +08:00
Chris An ce20ed4a4b Merge branch 'main' into fix/s3-post-object-location
Keep PostObjectLocationFixLayer on the gateway-only external HTTP stack,
ahead of ConditionalCorsLayer, and drop the legacy-stack Location
regression test now that the retired S3 stack selector is gone.
2026-11-19 16:43:56 +08:00
overtrue 24f3e783c2 fix(server): keep Connection off HTTP/2 answers and test the upload drain on the gateway stack
The three UploadPart timeout tests drove the early-response drain around
a hand-written s3s S3 service. They now drive it around the gateway
stack as production configures it (GatewayStack::assemble_serving, a
test-only constructor over the same configured builder), serving an
UploadPart handler that reads its whole body as RustFS's handler does.
Every assertion is kept: 400 RequestTimeout, the HTTP/1 transport
retained for the drain and drained of the late bytes, the HTTP/2 body
dropped alone, and other HTTP/2 and HTTP/3 streams still served.

The gateway times a stalled body out with its own request-body idle
bound, so that bound is what the fixture sets; it does not follow a
paused tokio clock, so the HTTP/1-and-HTTP/2 case runs in real time
(200 ms bound) instead of advancing a paused clock.

Kept as written, the HTTP/2 assertion failed: GatewayStack::call added
`Connection: close` to any answer whose connection intent is to close,
HTTP/2 included, where the header is forbidden and only the stream
ends. It is now added to HTTP/1 answers only.
2026-11-19 16:15:49 +08:00
overtrue 583df55d90 test(server): name the metadata route table case type for clippy 2026-11-19 16:15:49 +08:00
overtrue 670ae01f7a test(server): restore website, metadata listing and table isolation coverage on the gateway stack
The retired legacy metadata route carried the website-host and metadata=true listing case tables, and a test that its authorization attaches the table listing snapshot. Re-create both against the gateway stack, observing the listing a caller receives, and probe the native admin and control-plane hosts for the unsigned x-amz header refusal.
2026-11-19 16:15:49 +08:00
overtrue 3a1f0c09b3 fix(server): let the owned early-response drain take bodies the gateway leaves unread
The gateway stack was called through rustfs-gateway's tower adapter. Under the RustFS profile
that adapter wraps the request body, takes it back when the handler drops it unread, and reads
it in a task of its own spawned on the ambient runtime (unread_body.rs, Retention::settle). That
task took the body away from EarlyResponseBodyService before its abandon slot could see it, so
RustFS's own drain never ran: its task was no longer spawned under the HTTP/1-2 task owner,
graceful shutdown could report its children retired while a detached drain still held the
transport body, and the early-response drain metric and log went silent. On origin/main the
legacy stack was the default assembly, so the listener test never crossed the adapter.

GatewayStack::call now calls the service's own entry and keeps the adapter's connection
verdict (connection_intent_of). The inert drain_unread_request_bodies setting is removed.
EarlyResponseBodyService is again the only HTTP/1 unread-body drain, with Connection: close
and the configured idle bound.

The gateway-stack test that asserted the adapter's drain timeout now asserts that the stack
releases an unread body when it answers. The configured-timeout half moves to
gateway_rejected_http1_body_uses_the_configured_drain_timeout, which composes
EarlyResponseBodyService over the real gateway stack for a 1 s and a zero timeout.

Mutation: restoring the adapter call turns all three red:
http12_actual_early_body_drain_is_owned_until_release (the drain never starts),
rejected_http1_body_is_released_to_the_host_drain and
gateway_rejected_http1_body_uses_the_configured_drain_timeout.
2026-11-19 16:15:49 +08:00
overtrue eed50fc5ff test(e2e): run the differential against the only stack
Neither side of dual_stack_differential is started with the retired
stack selector any more, which the binary under test now refuses: both
sides run the binary under test, or side A runs the released reference
binary named by RUSTFS_E2E_REFERENCE_BINARY. RUSTFS_E2E_STACK_B is
gone; the injected-region control is unchanged.
2026-11-19 16:15:49 +08:00
overtrue e462fafb65 docs: describe the gateway stack as the only S3 stack in the CI gates and the changelog 2026-11-19 16:15:49 +08:00
overtrue 9c0bc265c3 ci: drop RUSTFS_S3_STACK from reference parity, nextest notes and the connect boundary guard 2026-11-19 16:15:49 +08:00
overtrue 74bf3ca5fa ci(nightly): run the dual-stack nightly on the gateway stack and archive the both-stack scripts 2026-11-19 16:15:49 +08:00
overtrue 31c4f4a7ff ci(acceptance): judge one server and drop the legacy reference run and the stack diff 2026-11-19 16:15:48 +08:00
overtrue 3fb8d861f0 ci(e2e): run the distributed, replication and migration-interop lanes on one stack 2026-11-19 16:15:48 +08:00
overtrue 4b386794e1 ci(mint): run one server on the gateway stack and drop the stack comparison 2026-11-19 16:15:48 +08:00
overtrue 23a2593aac ci: fold the gateway-stack lanes into the main lint, test and e2e jobs 2026-11-19 16:15:48 +08:00
overtrue 7b1befa6f9 chore(ci): shrink the s3s footprint to what the gateway-only stack keeps
server/layer.rs and server/hybrid.rs no longer reference s3s and leave
the edge allowlist; the baseline ceilings drop to the measured values
(files 50, s3_error! lines 103).
2026-11-19 16:15:48 +08:00
overtrue 2f3359df69 refactor(server): delete the five s3s patch layers the gateway stack replaces
SigV4HeaderGuard, S3ErrorMessageCompat, DoubleSlashListBucketsCompat,
VirtualHostStyleHint and ObjectAttributesEtagFix patched the s3s edge
and have no work left: the gateway_stack legacy_layers tests pin what
each made the legacy stack answer against the gateway stack alone. The
layer unit tests go with the layers, and the server-domain flag that
only gated VirtualHostStyleHint goes with it.

EmptyBodyContentLengthCompat keeps its admin, console and internode
halves and loses its S3 half, which the gateway stack replaces by
leaving the bodies of bodyless operations unread; its S3 tests now
assert that such requests pass through unchanged.
2026-11-19 16:15:48 +08:00
overtrue ea522c18b6 refactor(server): serve the outer tower stack with an owned body and error type
The S3 service under the middleware now answers in EdgeBody, a boxed
Bytes body, with a boxed error, instead of s3s::Body and
s3s::HttpError. The gateway stack, the native console and the refusals
written in server/http.rs meet in that one type; Swift and the
internode RPC lane convert into it from rustfs_s3_types::Body.
ControlBodyDigestService is generic over its inner response body, and
its control-plane XML body limit is a named constant equal to the limit
the s3s edge applied.
2026-11-19 16:15:48 +08:00
overtrue 9f13eb22a5 refactor(server)!: make the gateway stack the only S3 stack
Remove the S3Stack selector, the legacy s3s service assembly and the
gateway-stack cargo feature: the gateway stack, its native admin and
console hosts and the control-plane host are always assembled, and the
gateway crates are ordinary dependencies. The debug-only compile_error
guard goes with the feature.

RUSTFS_S3_STACK is retired: a server or an embedded server started with
it set, to any value including gateway or an empty string, refuses to
start with an error naming the variable and this release. The message
and its tests share one constant.

Tests that ran only on the legacy stack, or selected between stacks,
are retired; tests that ran on both now run on the gateway stack alone.
The s3s metadata/website route is no longer mounted and stays only
until the metadata-route branch replaces it.
2026-11-19 16:15:48 +08:00
overtrue fec2d4bb4b test: register the legacy-stack tests the gateway flip retires
Each entry names a legacy-only test, or the unit test of an s3s patch
layer, that the flip deletes, with the gateway-stack test that keeps
its contract (rustfs/backlog#2783). The commits that delete them delete
their entries.
2026-11-19 16:15:47 +08:00
overtrue 42a1417ed0 fix(notify): bind dispatch rules to bucket metadata snapshots 2026-11-19 10:15:02 +08:00
Chris An 7b7f96924b test: use typo-safe UTF-8 fixtures for POST Object Location
crate-ci typo tokenizes the caf prefix in caf%C3%A9 as a misspelling of
calf. Keep a UTF-8 percent-encoding case, but avoid that false positive.
2026-11-19 08:14:42 +08:00
overtrue a3d6b59d03 feat(server): answer website hosts on the gateway stack before S3 routing
The gateway selection gains a website service, chosen by its host claim
before the Keystone refusal, the native console, the native admin and the
control plane, as the legacy metadata route claims those requests first.
A parity test sends both request classes to both stacks over one store.
2026-11-19 07:44:46 +08:00
overtrue 924eb06968 feat(gateway): serve metadata=true listings on the gateway stack
Install the RustFS dialect rows of ListObjectsV2WithMetadata and
ListObjectVersionsWithMetadata on the S3 table, let the early gate allow
them, and register one handler that runs the shared access phase, the
listing authorization and the shared listing answer. The rows keep their
overlaps with the standard listings and each other; the admin extension
rows they also shadow live on the control-plane host.
2026-11-19 07:44:46 +08:00
overtrue 5a44944095 refactor(app): serve website hosts and metadata listings through owned modules
The legacy MetadataRoute now hands website requests to an owned WebsiteHost,
which reads through the owned GetObject/HeadObject authorization and use
cases with an anonymous request envelope, and hands metadata=true listings
to an owned module that authorizes and answers them on the request envelope.
The WebsiteRead marker moves with the website rules.
2026-11-19 07:44:45 +08:00
Chris An 4c869cd08c fix(server): emit absolute encoded POST Object Location
Browser POST uploads with success_action_status=201 currently return a
relative, unencoded Location such as /bucket/a+b c%d.txt. Clients that
decode the field the way AWS formats it treat + as a space.

Rebuild PostResponse Location as an absolute URI from the request host,
scheme, and addressing style, percent-encoding the object key with RFC
3986 path rules (space as %20, slash preserved as a separator).

Fixes #9259
2026-11-19 06:42:14 +08:00
168 changed files with 4495 additions and 9149 deletions
-2
View File
@@ -103,8 +103,6 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/test_retain_validation_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_ci_timing_report.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_junit_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_dual_stack.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_mint_stack_diff.py
$(RUSTFS_PYTHON_BIN) ./scripts/check_conformance_rulings.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/compat_matrix_report.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/s3-tests/test_report_compat.py
+2 -3
View File
@@ -723,9 +723,8 @@ test-group = 'ecstore-serial-flaky'
# fixture as the existing smoke modules.
#
# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
# starts two single-node servers from the same binary, one per
# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
# SDK script covering every operation of `impl S3 for FS` against both. Its
# starts two single-node servers on random ports with isolated temp
# dirs, and runs one SDK script covering every S3 operation against both. Its
# comparator unit tests need no server. It meets the admission criteria
# unchanged; the module clause admits a new differential test without another
# filter edit.
+2 -2
View File
@@ -9,9 +9,9 @@
# Adding a target is one more [[targets]] table, for example:
#
# [[targets]]
# name = "candidate-gateway"
# name = "candidate-tuned"
# binary = "candidate"
# env = { RUSTFS_S3_STACK = "gateway" }
# env = { RUSTFS_EXAMPLE_SETTING = "on" }
schema = 1
-2
View File
@@ -28,8 +28,6 @@ rustfs/src/app/storage_api.rs
rustfs/src/admin/router.rs
rustfs/src/app/metadata_route.rs
rustfs/src/server/http.rs
rustfs/src/server/layer.rs
rustfs/src/server/hybrid.rs
rustfs/src/protocols/client.rs
# e2e oracle: the test target that verifies S3 behaviour against s3s
+2 -2
View File
@@ -13,6 +13,6 @@
# s3_error_lines lines holding s3_error!, same scope, minus those the script
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=63
s3_error_lines=209
files=50
s3_error_lines=103
ecstore_files=0
-54
View File
@@ -1,54 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack leg
description: Prove which S3 stack one e2e leg's servers selected and upload the leg for the summary (call with if always)
inputs:
profile:
description: The nextest profile the leg ran
required: true
stack:
description: The stack the leg requested, legacy or gateway
required: true
logs:
description: Directory or file holding the servers' output
required: true
runs:
using: composite
steps:
- name: Prove which stack the servers selected
shell: bash
env:
PROFILE: ${{ inputs.profile }}
STACK: ${{ inputs.stack }}
LOGS: ${{ inputs.logs }}
run: |
# Outside the workspace: e2e_binary.py refuses a run whose source tree changed.
mkdir -p "$RUNNER_TEMP/leg"
junit="target/nextest/${PROFILE}/junit.xml"
if [ -s "$junit" ]; then cp "$junit" "$RUNNER_TEMP/leg/junit.xml"; fi
# A gateway leg stopped by its preflight has no output; the preflight's stack.json stays.
if [ -e "$LOGS" ]; then python3 scripts/dual_stack.py stack --expect "$STACK" --out "$RUNNER_TEMP/leg/stack.json" "$LOGS"; fi
- name: Upload the leg
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-${{ inputs.profile }}-${{ inputs.stack }}
overwrite: true
path: ${{ runner.temp }}/leg
if-no-files-found: warn
retention-days: 7
@@ -1,56 +0,0 @@
# Copyright 2026 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Both-stack summary
description: Diff the legacy and gateway legs of this run (scripts/junit_diff.py) and upload one dated summary
inputs:
profiles:
description: Comma-separated nextest profiles whose two legs this run uploaded
required: true
runs:
using: composite
steps:
- name: Download the legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: dual-stack-leg-*
path: legs
- name: Diff every profile and write the summary
id: summary
shell: bash
env:
PROFILES: ${{ inputs.profiles }}
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
date="$(date -u +%F)"
echo "date=${date}" >> "$GITHUB_OUTPUT"
mkdir -p legs
python3 scripts/dual_stack.py summary --profiles "$PROFILES" --legs legs --date "$date" --out-dir dual-stack
status=$?
if [ -f dual-stack/summary.md ]; then cat dual-stack/summary.md >> "$GITHUB_STEP_SUMMARY"; fi
exit "$status"
- name: Upload the dated summary
if: always() && steps.summary.outputs.date != ''
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-summary-${{ steps.summary.outputs.date }}
overwrite: true
path: dual-stack
if-no-files-found: error
retention-days: 30
+1 -1
View File
@@ -814,7 +814,7 @@
"monitor": "workflow",
"max_age_hours": 36,
"result_owner": "dual-stack-nightly.yml",
"disposition": "both-stack JUnit diff of the two profiles only ci.yml runs; the gateway legs are report-only until T2.10",
"disposition": "nightly run of the two e2e profiles only ci.yml runs, on the gateway stack, with the JUnit reports retained for 30 days",
"failure_owner": "watchdog",
"origin": "added"
},
+1 -173
View File
@@ -92,7 +92,6 @@ jobs:
connect: ${{ steps.scope.outputs.connect || 'true' }}
packages: ${{ steps.scope.outputs.packages }}
full_validation: ${{ steps.scope.outputs.full_validation || 'true' }}
gateway_smoke: ${{ steps.scope.outputs.gateway_smoke || 'true' }}
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
@@ -119,7 +118,7 @@ jobs:
echo "Base CI policy unavailable; running the full matrix."
fi
if ! grep -q '^plan=' "$GITHUB_OUTPUT" || ! grep -q '^plan_contract=8$' "$GITHUB_OUTPUT"; then
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'gateway_smoke=true' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
printf '%s\n' 'mode=full' 'connect=true' 'full_validation=true' 'packages=' 'plan=' >> "$GITHUB_OUTPUT"
echo "Base policy does not cover this workflow contract; retaining the full matrix."
fi
- name: Retain verification plan
@@ -676,54 +675,6 @@ jobs:
retention-days: 3
if-no-files-found: warn
test-and-lint-gateway-stack:
name: Test and Lint (gateway-stack)
# The gateway S3 stack (rustfs/backlog#2734) sits behind the rustfs
# `gateway-stack` feature, which no other pull-request lane lints or tests.
# Unlike the dormant feature lanes above, this one runs for every code
# change: each operation slice puts its handler tests and its refusal-matrix
# row behind the feature, and a lane that ran only on demand would let them
# rot behind a green pull request.
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
# Self-hosted for build scale and cache ownership: a workspace-scale Cargo
# build that reads the ci-dev cache written on this image.
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev-ea7eec74fe14
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# A feature that does not compile fails here; nothing below is skipped
# into a pass.
- name: Run clippy with gateway-stack
run: cargo clippy --locked -p rustfs --all-targets --features gateway-stack -- -D warnings
# The gateway stack's own tests: the assembly, the operation table and
# its completeness, the shared handler sequence, the stack selection, and
# the embedded server driven through the whole middleware stack. The
# filter names modules, never operations.
- name: Run tests with gateway-stack
env:
# The same bounded link concurrency as the other feature lanes.
CARGO_BUILD_JOBS: "2"
run: |
cargo nextest run --profile ci --locked -p rustfs --features gateway-stack --no-tests fail \
--status-level all --final-status-level all \
-E 'test(/gateway_stack|gateway_handlers|server::stack::/) | binary(embedded_gateway_stack_test)'
build-rustfs-debug-binary:
name: Build RustFS Debug Binary
if: contains(fromJSON('["full", "e2e"]'), needs.classify-changes.outputs.mode) && (github.event_name != 'pull_request' || github.event.action != 'closed')
@@ -986,8 +937,6 @@ jobs:
# Retain the producer image and fixture paths until the runtime pilot passes.
runs-on: sm-standard-4
timeout-minutes: 30
env:
RUSTFS_S3_STACK: legacy
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -1108,125 +1057,6 @@ jobs:
path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
retention-days: 3
# The same e2e-smoke suite against the same debug binary with the gateway
# S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
# Test and Lint aggregate and becomes required at T2.10. The test-binary
# download, identity, membership, run and completeness steps must stay
# identical to e2e-tests; scripts/ci_gate.py enforces that.
#
# When it runs: every push to main or gateway/integration, merge group,
# nightly schedule and manual dispatch, and every pull request that changes a
# gateway-stack input (GATEWAY_SMOKE_PATHS and the rustfs-gateway pins in
# scripts/ci_gate.py, published by Select CI scope as gateway_smoke).
# Why it is skipped on every other pull request: until T2.10 the gateway
# stack does not serve the whole smoke suite, so the lane fails on any
# revision. A pull request that touches none of its inputs can neither cause
# nor fix that failure, and a red check there says nothing about the change.
# Skipped is the honest state; a failure must never be rendered green, so no
# step here may carry continue-on-error. A selection that is missing or
# unreadable runs the lane. At T2.10, when the lane becomes required, it
# must run on every pull request again: delete the selection, do not extend
# it.
e2e-tests-gateway:
name: e2e-smoke (gateway, report-only until T2.10)
needs: [ classify-changes, build-rustfs-debug-binary ]
if: github.event_name != 'pull_request' || needs.classify-changes.outputs.gateway_smoke != 'false'
runs-on: sm-standard-2
timeout-minutes: 30
continue-on-error: true
env:
RUSTFS_S3_STACK: gateway
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Install verification runtime
uses: ./.github/actions/test-runtime
- name: Download E2E test binaries
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: e2e-test-binaries
path: ${{ runner.temp }}
- name: Verify E2E archive identity
run: python3 scripts/ci_test_artifact.py verify --configuration e2e --archive "$RUNNER_TEMP/e2e-tests.tar.zst"
- name: Set up Python
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Install awscurl
run: |
python3 -m pip install --user --upgrade pip "awscurl==0.44"
echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
- name: Verify awscurl
run: test -x "$AWSCURL_PATH"
# Download after the cache restore so the freshly built binary from the
# build job always wins over anything restored into target/debug.
- name: Download debug binary
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: rustfs-debug-binary
path: target/debug
- name: Make binary executable
run: chmod +x ./target/debug/rustfs
# Build the e2e test graph once. The archive is reused by the smoke
# selection guard, security exact-count check, and run below, avoiding a
# second compile of the same e2e_test target on cold runners (backlog#1645).
- name: Verify e2e smoke test membership
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
run: |
cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --workspace-remap "$GITHUB_WORKSPACE" --extract-to "$GITHUB_WORKSPACE" --extract-overwrite --message-format json > "${NEXTEST_LISTING}"
python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
# PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
# profile.e2e-smoke default-filter in .config/nextest.toml is the single
# wiring mechanism for e2e tests in CI — extend that filter instead of
# adding new e2e jobs here. Each test spawns its own rustfs server on a
# random port and reuses the downloaded debug binary above.
- name: Run e2e smoke suite
env:
NEXTEST_ARCHIVE: ${{ runner.temp }}/e2e-tests.tar.zst
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
SAMPLER_INTERVAL_SECS: "15"
run: |
./scripts/ci/resource_sampler.sh start e2e-smoke
trap './scripts/ci/resource_sampler.sh stop' EXIT
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --cargo-metadata target/nextest/cargo-metadata.json --binaries-metadata target/nextest/binaries-metadata.json --workspace-remap "$GITHUB_WORKSPACE" --target-dir-remap "$GITHUB_WORKSPACE/target" \
--status-level all --final-status-level all --failure-output final
- name: Verify e2e smoke execution completeness
run: python3 scripts/ci_test_artifact.py evidence "$RUNNER_TEMP/rustfs-e2e-smoke-list.json" target/nextest/e2e-smoke/junit.xml
- name: Upload e2e smoke diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}-${{ github.run_attempt }}
path: |
${{ runner.temp }}/rustfs-e2e-smoke-logs/
${{ runner.temp }}/rustfs-e2e-smoke-list.json
if-no-files-found: warn
- name: Upload e2e smoke JUnit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-smoke-gateway-junit-${{ github.run_number }}-${{ github.run_attempt }}
path: target/nextest/e2e-smoke/junit.xml
if-no-files-found: warn
e2e-full:
name: End-to-End Tests (full merge gate)
# Merge gate only (backlog#1149 ci-5): the never-automated user-visible
@@ -1612,7 +1442,6 @@ jobs:
- fips-provider
- test-and-lint-rio-v2
- test-and-lint-protocols
- test-and-lint-gateway-stack
- build-rustfs-debug-binary
- uring-integration
- e2e-tests
@@ -1648,7 +1477,6 @@ jobs:
- core-contract
- test-and-lint-rio-v2
- test-and-lint-protocols
- test-and-lint-gateway-stack
- build-rustfs-debug-binary
- build-rustfs-debug-binary-rio-v2
- uring-integration
+21 -68
View File
@@ -12,21 +12,18 @@
# See the License for the specific language governing permissions and
# limitations under the License.
# Nightly both-stack diff of the two e2e profiles only ci.yml runs, e2e-smoke
# and e2e-full (rustfs/backlog#2772, T3.1). The other profiles carry a `stack`
# dimension in their own workflows, where the existing lane is the legacy leg.
# Nightly run of the two e2e profiles only ci.yml runs, e2e-smoke and e2e-full
# (rustfs/backlog#2772, T3.1), on the gateway stack, which is the only S3 stack.
# The workflow keeps its file and registry name from the period when it diffed two
# stacks. Each leg uploads its JUnit report and server logs for 30 days; the one
# comparison against the last legacy record (the `dual-stack-leg-<profile>-legacy`
# artifacts of the final both-stack run) is made by hand with scripts/junit_diff.py.
#
# One build serves every leg. One preflight starts that binary with
# RUSTFS_S3_STACK=gateway (scripts/dual_stack.py preflight); without a proven
# gateway start the gateway legs do not run and the summary says `skipped:
# gateway stack refuses startup: <why>`. Which stack a leg ran is read from the
# servers' own start-up line. The gateway legs are report-only until T2.10.
#
# Left out of both stacks: `dual_stack_differential::` (servers with an explicit
# Left out: `dual_stack_differential::` (servers with an explicit
# stack), `fake_s3_target::`, the upgrade write-readiness cases and one deadline
# case (no server), and `distributed_startup_regression_test::` (needs ci.yml's
# hooks-binary manifest). Both e2e-full stacks use ci.yml's pinned mc and
# Vault binaries and network fault-injection tools.
# hooks-binary manifest). e2e-full uses ci.yml's pinned mc and Vault binaries
# and network fault-injection tools.
name: dual-stack-nightly
@@ -50,13 +47,11 @@ defaults:
jobs:
build:
name: Build once and probe the gateway stack
name: Build once
if: github.repository == 'rustfs/rustfs'
# The runner and cache of ci.yml's build-rustfs-debug-binary.
runs-on: sm-standard-4
timeout-minutes: 60
outputs:
stacks: ${{ steps.preflight.outputs.stacks }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -86,47 +81,15 @@ jobs:
if-no-files-found: error
retention-days: 1
# Without a proven gateway start there is no gateway leg to run.
- name: Gateway preflight
id: preflight
run: |
if python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/preflight/stack.json"; then
echo 'stacks=["legacy","gateway"]' >> "$GITHUB_OUTPUT"
else
echo 'stacks=["legacy"]' >> "$GITHUB_OUTPUT"
fi
# The evidence stands in for a gateway leg that is skipped; a leg that runs overwrites it.
- name: Upload the preflight evidence for e2e-smoke
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-smoke-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
- name: Upload the preflight evidence for e2e-full
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: dual-stack-leg-e2e-full-gateway
path: ${{ runner.temp }}/preflight
if-no-files-found: warn
retention-days: 7
leg:
name: ${{ matrix.profile }} (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: ${{ matrix.profile }}
needs: [ build ]
runs-on: sm-standard-4
timeout-minutes: 120
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
profile: [ e2e-smoke, e2e-full ]
stack: ${{ fromJSON(needs.build.outputs.stacks) }}
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -208,10 +171,9 @@ jobs:
run: |
"$RUSTFS_TEST_VAULT_BIN" version
- name: Run the profile on the ${{ matrix.stack }} stack
- name: Run the profile
env:
PROFILE: ${{ matrix.profile }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/server-logs
run: |
chmod +x target/debug/rustfs
@@ -220,23 +182,14 @@ jobs:
filter="$filter & not test(=list_objects_v2_pagination_test::tests::test_list_objects_v2_deadline_rejects_unresponsive_peer)"
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile "$PROFILE" -p e2e_test -E "$filter"
- name: Prove the stack and upload the leg
- name: Upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with: { profile: "${{ matrix.profile }}", stack: "${{ matrix.stack }}", logs: "${{ runner.temp }}/server-logs" }
summary:
name: Both-stack diff summary
needs: [ build, leg ]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-smoke,e2e-full
name: e2e-nightly-${{ matrix.profile }}
overwrite: true
path: |
target/nextest/${{ matrix.profile }}/junit.xml
${{ runner.temp }}/server-logs
if-no-files-found: warn
retention-days: 30
+2 -33
View File
@@ -53,14 +53,11 @@ concurrency:
jobs:
distributed:
if: github.repository == 'rustfs/rustfs'
name: Distributed 4-node 4-disk e2e (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Distributed 4-node 4-disk e2e
# GitHub-hosted VM: loop and tmpfs mounts work here. sm-standard-4 is an
# ARC pod and rejects both (`mount -o loop` ENOENT, tmpfs "read-only").
runs-on: ubuntu-latest
timeout-minutes: 180
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
NO_PROXY: 127.0.0.1,localhost
@@ -132,11 +129,6 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify distributed e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -148,7 +140,6 @@ jobs:
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-distributed-logs
FILTER: ${{ inputs.filter }}
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: |
set -euo pipefail
rm -f target/nextest/e2e-distributed/junit.xml
@@ -159,19 +150,11 @@ jobs:
fi
test -s target/nextest/e2e-distributed/junit.xml
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-distributed
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-distributed-logs
- name: Upload distributed e2e diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-distributed-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-distributed-${{ github.run_number }}
path: |
target/nextest/e2e-distributed/junit.xml
${{ runner.temp }}/rustfs-e2e-distributed-list.json
@@ -191,20 +174,6 @@ jobs:
fi
done
dual-stack-summary:
name: Both-stack diff summary
needs: [distributed]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-distributed
alert-on-failure:
name: Alert on scheduled failure
needs: [distributed]
+8 -76
View File
@@ -54,18 +54,15 @@ concurrency:
jobs:
repl-nightly:
if: github.repository == 'rustfs/rustfs'
name: Replication e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Replication e2e (nightly)
# dual-node tests spawn two full rustfs servers each; the extra headroom
# over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
# from starving the runner. Mirrors the ILM serial lane's runner class.
runs-on: sm-standard-4
timeout-minutes: 45
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'gateway-stack' || '' }}
RUSTFS_BUILD_FEATURES: ""
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -99,11 +96,6 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify replication e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
@@ -114,22 +106,13 @@ jobs:
- name: Run replication e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run --profile e2e-repl-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-repl-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
- name: Upload nextest junit report
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-replication-nightly-junit-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-replication-nightly-junit-${{ github.run_number }}
path: |
target/nextest/e2e-repl-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
@@ -139,12 +122,9 @@ jobs:
cluster-nightly:
if: github.repository == 'rustfs/rustfs'
name: Cluster fault e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Cluster fault e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
@@ -165,11 +145,6 @@ jobs:
run: |
python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
- name: Verify cluster fault e2e membership
env:
NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -181,22 +156,13 @@ jobs:
env:
CARGO_BIN_EXE_rustfs: ${{ github.workspace }}/target/debug/rustfs
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
run: python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-nightly -p e2e_test
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-nightly
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-nightly-logs
- name: Upload cluster fault diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-cluster-nightly-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-cluster-nightly-${{ github.run_number }}
path: |
target/nextest/e2e-nightly/junit.xml
${{ runner.temp }}/rustfs-e2e-nightly-list.json
@@ -206,15 +172,12 @@ jobs:
protocols-nightly:
if: github.repository == 'rustfs/rustfs'
name: Protocol e2e (nightly, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: Protocol e2e (nightly)
runs-on: sm-standard-4
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
RUSTFS_BUILD_FEATURES: ${{ matrix.stack == 'gateway' && 'ftps,webdav,sftp,gateway-stack' || 'ftps,webdav,sftp' }}
RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -238,11 +201,6 @@ jobs:
- name: Build protocol server
run: python3 scripts/e2e_binary.py build --features "$RUSTFS_BUILD_FEATURES"
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The suite owns fixed protocol ports and serializes its internal cases.
- name: Verify protocol e2e membership
env:
@@ -254,28 +212,16 @@ jobs:
- name: Run protocol e2e nightly suite
env:
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
RUSTFS_S3_STACK: ${{ matrix.stack }}
# These servers set no log level of their own; the stack proof reads their start-up line.
RUST_LOG: rustfs=info
# The servers inherit nextest's stdout (--no-capture): keep it where the stack proof reads it.
run: |
set -o pipefail
mkdir -p "$RUSTFS_E2E_LOG_DIR"
python3 scripts/e2e_binary.py run --features "$RUSTFS_BUILD_FEATURES" -- cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture 2>&1 | tee "$RUSTFS_E2E_LOG_DIR/nextest.log"
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-protocols
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-protocol-e2e-logs
- name: Upload protocol diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: e2e-protocol-nightly-${{ github.run_number }}-${{ matrix.stack }}
name: e2e-protocol-nightly-${{ github.run_number }}
path: |
target/nextest/e2e-protocols/junit.xml
${{ runner.temp }}/rustfs-e2e-protocols-list.json
@@ -283,20 +229,6 @@ jobs:
retention-days: 7
if-no-files-found: warn
dual-stack-summary:
name: Both-stack diff summary
needs: [repl-nightly, cluster-nightly, protocols-nightly]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-repl-nightly,e2e-nightly,e2e-protocols
alert-on-failure:
name: Alert on scheduled failure
needs: [repl-nightly, cluster-nightly, protocols-nightly]
+19 -78
View File
@@ -15,31 +15,20 @@
# Gateway black-box acceptance lanes (rustfs/backlog#2775 T3.4, #2776 T3.5).
#
# Runs two suites of the rustfs/gateway repository against the candidate binary,
# once per S3 stack (RUSTFS_S3_STACK=legacy and =gateway), one server per stack:
# one server:
#
# - the conformance corpus, `--profile rustfs`. That profile describes the
# gateway edge, so scripts/check_conformance_rulings.py judges the gateway
# stack against .config/conformance-rulings.toml (red: an unruled failure, a
# ruled case failing differently, a stale ruling, a skip on the server's own
# answer) and publishes the legacy run as a reference that is never judged:
# it only has to serve, stay up and complete;
# - the client matrix, ci/compat/run_matrix.sh --external, judged per stack by
# - the conformance corpus, `--profile rustfs`, judged by
# scripts/check_conformance_rulings.py against .config/conformance-rulings.toml
# (red: an unruled failure, a ruled case failing differently, a stale ruling,
# a skip on the server's own answer);
# - the client matrix, ci/compat/run_matrix.sh --external, judged by
# scripts/compat_matrix_report.py (no `fail`, no `sut-unregistered`, every
# client ran; a client that could not be installed is reported as skipped).
# The legacy matrix is the baseline: every cell that passes on legacy must
# pass on gateway (`compare`).
#
# The suites are checked out at the rustfs/gateway rev Cargo.toml pins, so the
# suite and the framework under test never drift apart. The verdict of each
# stack is computed from what the run left on disk (startup record, the stack
# the server's own log names, runner exit code, liveness after the corpus,
# report), never from the exit status of an earlier step. Needs
# scripts/dual_stack.py and the `s3_stack` startup field of rustfs/rustfs#9367.
#
# The gateway stack is report-only until T2.10: its job may fail without failing
# the workflow, and its summary says what it observed (did not start, or how many
# cases ran, passed, failed and were answered 501). The stack comparison is
# report-only for the same reason; its conformance half is information only.
# suite and the framework under test never drift apart. The verdict is computed
# from what the run left on disk (startup record, runner exit code, liveness
# after the corpus, report), never from the exit status of an earlier step.
#
# Not a pull-request gate. Scheduled failures are owned by the scheduled-validation
# watchdog (.github/scheduled-validations.json).
@@ -166,21 +155,15 @@ jobs:
retention-days: 1
acceptance:
name: ${{ matrix.stack == 'gateway' && 'Acceptance (gateway, report-only until T2.10)' || 'Acceptance (legacy)' }}
name: Acceptance
needs: [ plan, build-candidate, build-gateway-tools ]
# GitHub-hosted like the s3-tests targets of reference-parity.yml: the job
# only runs prebuilt binaries and the pinned clients, none of which needs Docker.
runs-on: ubuntu-latest
# About 3x the 15 minutes the legacy job took in run 43325101234 (client install about
# About 3x the 15 minutes the job took in run 43325101234 (client install about
# 8, matrix about 3, corpus about 3), so a hung server costs 45 runner minutes, not more.
timeout-minutes: 45
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [ legacy, gateway ]
env:
STACK: ${{ matrix.stack }}
ENDPOINT: http://127.0.0.1:9000
# The conformance suite signs with these fixed fixture credentials (the AWS
# documentation example pair), so the candidate is started with them and the
@@ -213,10 +196,8 @@ jobs:
name: gateway-acceptance-tools
path: tools
# Records `ready`, or what the process said when it exited, in lane/startup.txt. `ready`
# also needs the server's own `server_starting` log event to name the requested stack
# (its `s3_stack` field comes from rustfs/rustfs#9367; scripts/dual_stack.py reads it).
- name: Start candidate (${{ matrix.stack }} stack)
# Records `ready`, or what the process said when it exited, in lane/startup.txt.
- name: Start candidate
run: |
set -uo pipefail
chmod +x target/debug/rustfs tools/*
@@ -224,7 +205,7 @@ jobs:
target/debug/rustfs --version | tee lane/candidate-version.txt
# The four directories share the runner's one disk; the bypass is the CI case the
# physical-disk guard sanctions (as in mint.yml).
RUSTFS_S3_STACK="$STACK" RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true RUST_LOG=info target/debug/rustfs --address 127.0.0.1:9000 \
RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true RUST_LOG=info target/debug/rustfs --address 127.0.0.1:9000 \
--access-key "$ACCESS_KEY" --secret-key "$SECRET_KEY" "$RUNNER_TEMP"/data/rustfs{0..3} \
> lane/rustfs.log 2>&1 &
pid=$!
@@ -239,13 +220,8 @@ jobs:
exit 1
fi
if curl -sf "$ENDPOINT/health/ready" >/dev/null; then
if python3 scripts/dual_stack.py stack --expect "$STACK" --out lane/stack.json lane/rustfs.log > /dev/null; then
echo ready > lane/startup.txt
exit 0
fi
echo "did not start: stack not observed" > lane/startup.txt
cat lane/startup.txt
exit 1
echo ready > lane/startup.txt
exit 0
fi
sleep 1
done
@@ -338,58 +314,23 @@ jobs:
run: |
set -uo pipefail
status=0
python3 scripts/check_conformance_rulings.py --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/compat_matrix_report.py check --stack "$STACK" --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/check_conformance_rulings.py --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
python3 scripts/compat_matrix_report.py check --run-dir lane | tee -a "$GITHUB_STEP_SUMMARY" || status=1
exit "$status"
- name: Upload lane results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: gateway-acceptance-lane-${{ matrix.stack }}
name: gateway-acceptance-lane
overwrite: true
path: |
lane/*.txt
lane/*.log
lane/report.json
lane/stack.json
lane/compat/matrix.json
lane/compat/probe.jsonl
lane/compat/sut.log
lane/compat/results/**
if-no-files-found: error
retention-days: 14
compare:
name: Compare stacks (report-only until T2.10)
needs: [ acceptance ]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 10
continue-on-error: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download lane results
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: gateway-acceptance-lane-*
path: lanes
- name: Diff legacy and gateway
run: |
set -uo pipefail
status=0
{
python3 scripts/check_conformance_rulings.py \
--diff lanes/gateway-acceptance-lane-legacy lanes/gateway-acceptance-lane-gateway || status=1
echo
python3 scripts/compat_matrix_report.py diff \
--legacy lanes/gateway-acceptance-lane-legacy --gateway lanes/gateway-acceptance-lane-gateway || status=1
} > diff.md
cat diff.md
cat diff.md >> "$GITHUB_STEP_SUMMARY"
exit "$status"
+2 -52
View File
@@ -44,13 +44,6 @@
# well inside the 120-minute budget and buys deterministic infra. The
# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
# has fewer cores and reintroduces fleet-state risk for no reliability gain.
#
# Stack dimension (rustfs/backlog#2774, T3.3): one image, run with RUSTFS_S3_STACK=legacy and
# =gateway. `compare` (scripts/mint_stack_diff.py) holds the gate that every case legacy passes
# passes on gateway with no new failure. The gateway leg and `compare` are report-only until
# T2.10. A gateway server that refuses to start stops its leg at "Wait for RustFS ready", before
# mint runs; the summary says `skipped: gateway stack refuses startup: <why>`. The stack is read
# from the server's own start-up line (scripts/dual_stack.py stack).
name: mint
@@ -145,16 +138,8 @@ jobs:
retention-days: 1
mint:
name: mint (${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: mint
needs: [image]
# Report-only until T2.10: the gateway stack cannot serve the suites yet.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy:
fail-fast: false
matrix:
stack: [legacy, gateway]
env:
STACK: ${{ matrix.stack }}
# GitHub-hosted: reliably provides a running Docker daemon + buildx +
# python3. See the header note (ci-2) for why the self-hosted sm-standard-4
# label was abandoned (no docker.sock on the pod its only run landed on).
@@ -194,7 +179,6 @@ jobs:
-e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
-e RUSTFS_S3_STACK="${STACK}" \
-e RUST_LOG=rustfs=info \
-v /tmp/rustfs-mint:/data \
rustfs-ci
@@ -291,47 +275,13 @@ jobs:
mkdir -p artifacts/rustfs-mint
docker logs rustfs-mint > artifacts/rustfs-mint/rustfs.log 2>&1 || true
- name: Prove which stack the server selected
if: always()
run: python3 scripts/dual_stack.py stack --expect "${STACK}" --out artifacts/stack.json artifacts/rustfs-mint
- name: Upload artifacts
if: always() && env.ACT != 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: mint-${{ matrix.stack }}
name: mint
path: artifacts/**
compare:
name: Compare stacks (report-only until T2.10)
needs: [mint]
if: always()
continue-on-error: true
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download both legs
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
pattern: mint-{legacy,gateway}
path: legs
# A leg that never uploaded leaves no directory: create both so that the
# comparison reports "no result" for it instead of failing to find it.
- name: Compare the stacks
run: |
# The shell runs with -e; the exit status is reported after the summary is written.
set +e -u -o pipefail
mkdir -p legs/mint-legacy legs/mint-gateway
python3 scripts/mint_stack_diff.py legs/mint-legacy legs/mint-gateway | tee report.txt
status="${PIPESTATUS[0]}"
{ echo '## Mint: gateway versus legacy'; echo; echo '```'; cat report.txt; echo '```'; } >> "$GITHUB_STEP_SUMMARY"
exit "$status"
alert-on-failure:
name: Alert on scheduled failure
needs: [image, mint]
@@ -68,14 +68,11 @@ env:
jobs:
minio-source:
name: MinIO source (read-through, list-through, backfill, ${{ matrix.stack }}${{ matrix.stack == 'gateway' && ', report-only until T2.10' || '' }})
name: MinIO source (read-through, list-through, backfill)
# Keep this scheduled interoperability lane scoped to the upstream repo.
if: github.repository == 'rustfs/rustfs'
runs-on: ubuntu-latest
timeout-minutes: 90
# The gateway leg is report-only until T2.10.
continue-on-error: ${{ matrix.stack == 'gateway' }}
strategy: { fail-fast: false, matrix: { stack: [legacy, gateway] } }
env:
NO_PROXY: 127.0.0.1,localhost
# Fixed credentials of the container this job starts and throws away;
@@ -140,11 +137,6 @@ jobs:
- name: Build the RustFS binary under test
run: python3 scripts/e2e_binary.py build --bins
# A gateway binary that refuses to start skips the suite; the summary says why.
- name: Gateway preflight
if: matrix.stack == 'gateway'
run: python3 scripts/dual_stack.py preflight --binary target/debug/rustfs --out "$RUNNER_TEMP/leg/stack.json"
# The lane selects tests by module, so a rename would quietly shrink it.
# The committed digest in .config/e2e-odm-interop-selection.txt fails
# closed on that.
@@ -155,18 +147,9 @@ jobs:
- name: Run the interop cases against MinIO
env:
RUSTFS_S3_STACK: ${{ matrix.stack }}
RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-logs
run: python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test --no-tests=fail
- name: Prove the stack and upload the leg
if: always()
uses: ./.github/actions/dual-stack-leg
with:
profile: e2e-odm-interop
stack: ${{ matrix.stack }}
logs: ${{ runner.temp }}/rustfs-e2e-logs
- name: Build the MinIO interop report
if: always()
uses: ./.github/actions/odm-interop-report
@@ -190,7 +173,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}-${{ matrix.stack }}
name: odm-interop-minio-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/odm-interop/minio
target/nextest/e2e-odm-interop/junit.xml
@@ -316,20 +299,6 @@ jobs:
target/nextest/e2e-odm-interop/junit.xml
retention-days: 14
dual-stack-summary:
name: Both-stack diff summary
needs: [minio-source]
if: github.repository == 'rustfs/rustfs' && always()
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: ./.github/actions/dual-stack-summary
with:
profiles: e2e-odm-interop
alert-on-failure:
name: Alert on scheduled failure
needs: [minio-source, cloud-source]
+4 -4
View File
@@ -32,9 +32,9 @@
#
# The `differential` job adds the 73-operation dual-stack differential
# (crates/e2e_test/src/dual_stack_differential, rustfs/backlog#2740) against the
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY,
# started without RUSTFS_S3_STACK), side B is the candidate built above on its
# legacy stack. It reuses the candidate build artifact, never builds the candidate
# same pinned reference: side A is the released binary (RUSTFS_E2E_REFERENCE_BINARY),
# side B is the candidate built above.
# It reuses the candidate build artifact, never builds the candidate
# twice, and prints reference_side, version_a and version_b from the report the test
# wrote; it fails when side A was not the reference or both sides report the same
# version. Every (op, field, a, b) difference goes to the job summary. Report-only
@@ -310,7 +310,7 @@ jobs:
chmod +x target/debug/rustfs
test -x "$REFERENCE_PATH"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
# Side A: the released binary. Side B: the candidate on its legacy stack.
# Side A: the released binary. Side B: the candidate.
export RUSTFS_E2E_REFERENCE_BINARY="$REFERENCE_PATH"
test_name='dual_stack_differential::tests::both_stacks_answer_every_operation_identically'
python3 scripts/e2e_binary.py run --features e2e-test-hooks -- \
+2
View File
@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Deployment
- **The S3 protocol stack is now rustfs/gateway**, and it is the only S3 stack. The legacy s3s-based stack is removed, and so are the `RUSTFS_S3_STACK` environment variable and the `gateway-stack` cargo feature. There is no on-disk format change. `RUSTFS_S3_STACK` no longer selects anything: a server started with it set, to any value, refuses to start and prints `RUSTFS_S3_STACK is no longer supported; the gateway stack is the only S3 stack since <version>`, where `<version>` is the release's version. Remove the variable from service units, container environments, Helm values and orchestration files before upgrading. To roll back, run the previous release binary on the same data.
- Before upgrading, verify that the configured root access key is at least 3 bytes and the root secret key is at least 8 bytes. RustFS now rejects shorter values at startup and identifies the setting to update. Version 1.0.5 accepted these settings at startup and returned HTTP 500 when root requested STS credentials. Coordinate root credential changes across all nodes and clients. See [root credential rotation considerations](docs/operations/two-factor-auth.md#root-credentials-cannot-be-changed-at-runtime).
- Existing volumes created by older root-running container images must be writable by the non-root runtime identity before upgrading. The default container UID/GID is `10001:10001`; stop the affected instances, verify host mount paths, and for ordinary bind mounts run `sudo chown -R 10001:10001 /actual/data /actual/logs` on each RustFS data/log mount. Account for ACLs, NFS, Kubernetes, and rootless UID mappings. Image-build ownership changes do not migrate mounted data. See the Docker Quick Start in [README.md](README.md).
@@ -171,70 +171,6 @@ impl DataUsageBucketObservation {
}
impl DataUsageInfo {
/// Hide values that cannot be bound to the current storage-owned identity.
///
/// `incarnations` contains canonical non-nil identities read from storage;
/// a missing entry means identity was unavailable. Neither legacy values
/// without identity nor metadata read failures establish confirmed zero.
/// Changed responses are display-only and cannot authorize quota or writes.
/// If retained counters overflow, all values remain unknown.
/// Returns the number of canonical values removed.
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
let original_count = self.buckets_usage.len();
let original_observation_count = self.usage_snapshot_bucket_observations.len();
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
incarnations.get(bucket).is_some_and(|current| {
!current.is_empty()
&& current != "00000000-0000-0000-0000-000000000000"
&& usage.bucket_incarnation.as_ref() == Some(current)
})
};
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
matches_current(bucket, &observation.usage)
&& observation
.sources
.iter()
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
});
if original_count == self.buckets_usage.len()
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
{
return 0;
}
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
self.buckets_usage.clear();
self.usage_snapshot_bucket_observations.clear();
[0; 4]
});
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_admin_projection = true;
original_count.saturating_sub(self.buckets_usage.len())
}
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
@@ -321,110 +321,3 @@ fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
#[test]
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
let mut candidate = snapshot();
let original = candidate.clone();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
assert_eq!(candidate, original);
assert!(candidate.is_complete_bucket_usage_snapshot());
}
#[test]
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
for identity in [
None,
Some(""),
Some("corrupt"),
Some("00000000-0000-0000-0000-000000000000"),
Some("22222222-2222-4222-8222-222222222222"),
] {
let mut candidate = snapshot();
candidate.usage_snapshot_bucket_observations.clear();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = identity.map(str::to_string);
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
1,
"identity {identity:?}"
);
assert!(!candidate.buckets_usage.contains_key("bucket"));
assert!(!candidate.bucket_sizes.contains_key("bucket"));
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
assert!(!candidate.is_valid_partial_snapshot());
}
for current in ["", "00000000-0000-0000-0000-000000000000"] {
let mut candidate = snapshot();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some(current.to_string());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
1
);
assert!(candidate.usage_snapshot_admin_projection);
}
let mut candidate = snapshot();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(7));
candidate.buckets_count = 2;
candidate.bucket_sizes.insert("cold".to_string(), 7);
candidate.calculate_totals();
candidate.replication_info.insert("target".to_string(), Default::default());
candidate.tier_stats = Some(Default::default());
candidate.unknown_tier_stats = Some(Default::default());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
1
);
assert_eq!(candidate.buckets_usage.len(), 1);
assert_eq!(candidate.buckets_usage["cold"].size, 7);
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
assert_eq!(
(
candidate.buckets_count,
candidate.objects_total_size,
candidate.objects_total_count,
candidate.versions_total_count
),
(1, 7, 7, 7)
);
assert!(candidate.replication_info.is_empty());
assert!(candidate.tier_stats.is_none());
assert!(candidate.unknown_tier_stats.is_none());
assert!(candidate.usage_snapshot_admin_projection);
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_overflow_leaves_totals_unknown() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
candidate.buckets_usage.insert("small".to_string(), usage(1));
candidate.buckets_count = 3;
let current = HashMap::from([
("large".to_string(), INCARNATION.to_string()),
("small".to_string(), INCARNATION.to_string()),
]);
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&current), 3);
assert!(candidate.buckets_usage.is_empty());
assert!(candidate.bucket_sizes.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
}
+1 -1
View File
@@ -29,7 +29,7 @@ Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
| **distributed 4×4** | [`src/distributed/`](src/distributed) | Nightly `e2e-distributed` lane: S3, object lock/WORM, versioning, bucket/site replication, quota, expand/decommission/rebalance, concurrency, chaos, 4-node upgrade of historical data and IAM AK/SK. Map: [`docs/testing/distributed-e2e.md`](../../docs/testing/distributed-e2e.md) |
| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
| **upgrade compatibility** | `upgrade_compatibility_test` | Pinned previous-release writes followed by current-build reads on the same data directory |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Side A is the binary under test on `RUSTFS_S3_STACK=legacy`; side B runs `RUSTFS_E2E_STACK_B`. Set `RUSTFS_E2E_REFERENCE_BINARY` to a released `rustfs` binary to make side A that binary instead (started without `RUSTFS_S3_STACK`, which older releases do not know). A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
| **dual-stack differential** | `dual_stack_differential` | Two single-node servers answer one SDK script for all 73 S3 operations and every differing `(op, field, a, b)` is reported. Both sides run the binary under test on its only (gateway) stack unless `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary, which then runs side A. A missing, non-file or non-executable path fails the run; it never skips or falls back to the same-binary mode. `RUSTFS_E2E_STACK_B_REGION` starts side B in another region to inject a difference. The report in `<target>/dual-stack/<run>.json` records `reference_side` and each side's `--version` output |
The external-tool `storage_metric_ownership_test` validates the OTLP/Collector/Prometheus path, including a rolling upgrade and node failures. See the [storage metrics guide](../../docs/operations/storage-metrics.md) for its required binaries and focused command.
@@ -12,14 +12,15 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
//! Differential: two `rustfs` processes, one SDK script (rustfs/backlog#2740).
//!
//! Stack A runs `RUSTFS_S3_STACK=legacy` of the binary under test, unless `RUSTFS_E2E_REFERENCE_BINARY`
//! names a released binary: then A is that binary, started without `RUSTFS_S3_STACK` (older
//! releases do not know it), and a bad path fails the run instead of falling back (see
//! `reference`). The report names which side is the reference and the `--version` each side ran. Stack B runs the value of `RUSTFS_E2E_STACK_B`
//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
//! set, starts B in another region so that a difference can be injected by hand. Both servers
//! Side B is the binary under test. Side A is the binary under test too, unless
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released binary: then A is that binary, and a bad path
//! fails the run instead of falling back (see `reference`). The report names which side is the
//! reference and the `--version` each side ran. `RUSTFS_E2E_STACK_B_REGION`, when set, starts B
//! in another region so that a difference can be injected by hand. Neither side is given the
//! retired stack selector: the binary under test refuses to start with it, and the gateway stack
//! is its only S3 stack. Both servers
//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
@@ -37,20 +38,18 @@ use std::time::{SystemTime, UNIX_EPOCH};
type BoxError = Box<dyn std::error::Error + Send + Sync>;
const STACK_A: &str = "legacy";
/// Label of a side that runs the binary under test.
const UNDER_TEST: &str = "under-test";
/// Label of side A in the report when it is the released reference binary.
const STACK_A_REFERENCE: &str = "reference";
const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
const DEFAULT_STACK_B: &str = "legacy";
/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
/// report its own name and would add CreateBucket differences to the control.
const CONTROL_REGION_A: &str = "us-west-2";
const CONTROL_REGION_B: &str = "eu-west-1";
/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
/// `normal` cases side A must answer with a 2xx. The comparison alone cannot tell a
/// script that exercises real behavior from one in which both stacks fail the same way. Five of
/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
@@ -60,7 +59,6 @@ const MIN_NORMAL_SUCCESSES: usize = 68;
struct PairConfig {
side_a: SideA,
stack_b: String,
region_a: Option<String>,
region_b: Option<String>,
}
@@ -69,7 +67,6 @@ impl PairConfig {
fn from_env() -> Result<Self, BoxError> {
Ok(Self {
side_a: SideA::from_env()?,
stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
region_a: None,
region_b: std::env::var(ENV_STACK_B_REGION).ok(),
})
@@ -77,16 +74,16 @@ impl PairConfig {
fn stack_a_label(&self) -> &'static str {
match self.side_a {
SideA::SameBinary => STACK_A,
SideA::SameBinary => UNDER_TEST,
SideA::Reference(_) => STACK_A_REFERENCE,
}
}
}
/// Starts one server of the binary under test on `stack`, overriding its region only when asked.
async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
/// Starts one server of the binary under test, overriding its region only when asked.
async fn start_under_test(region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = vec![("RUSTFS_S3_STACK", stack)];
let mut vars = Vec::new();
if let Some(region) = region {
vars.push(("RUSTFS_REGION", region));
}
@@ -94,8 +91,7 @@ async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvi
Ok(env)
}
/// Starts one server of a released binary. `RUSTFS_S3_STACK` is not passed: older releases do not
/// know it and have exactly one stack.
/// Starts one server of a released binary.
async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
let mut env = RustFSTestEnvironment::new().await?;
let mut vars = Vec::new();
@@ -111,11 +107,11 @@ async fn start_reference(binary: &Path, region: Option<&str>) -> Result<RustFSTe
async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
let side_a = async {
match &config.side_a {
SideA::SameBinary => start_stack(STACK_A, config.region_a.as_deref()).await,
SideA::SameBinary => start_under_test(config.region_a.as_deref()).await,
SideA::Reference(binary) => start_reference(binary, config.region_a.as_deref()).await,
}
};
tokio::try_join!(side_a, start_stack(&config.stack_b, config.region_b.as_deref()))
tokio::try_join!(side_a, start_under_test(config.region_b.as_deref()))
}
fn report_directory() -> PathBuf {
@@ -160,7 +156,7 @@ async fn run_differential(label: &str, config: &PairConfig) -> Run {
schema: 2,
run: format!("{label}-{millis}-{}", std::process::id()),
stack_a: config.stack_a_label().to_owned(),
stack_b: config.stack_b.clone(),
stack_b: UNDER_TEST.to_owned(),
reference_side: matches!(config.side_a, SideA::Reference(_)).then_some("a"),
version_a,
version_b,
@@ -235,7 +231,6 @@ mod tests {
init_logging();
let config = PairConfig {
side_a: SideA::SameBinary,
stack_b: STACK_A.to_owned(),
region_a: Some(CONTROL_REGION_A.to_owned()),
region_b: Some(CONTROL_REGION_B.to_owned()),
};
@@ -15,7 +15,7 @@
//! Which binary side A of the differential runs, and what it reports about itself.
//!
//! `RUSTFS_E2E_REFERENCE_BINARY` names a released `rustfs` binary to use as the reference. Unset,
//! side A is the binary under test on `RUSTFS_S3_STACK=legacy`. Set, it must resolve to an
//! side A is the binary under test. Set, it must resolve to an
//! executable file or the run fails: a reference that was asked for and silently replaced by the
//! binary under test would report an agreement with itself as agreement with the release.
@@ -28,9 +28,9 @@ pub(super) const ENV_REFERENCE_BINARY: &str = "RUSTFS_E2E_REFERENCE_BINARY";
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) enum SideA {
/// Both sides are the binary under test; A runs `RUSTFS_S3_STACK=legacy`.
/// Both sides are the binary under test.
SameBinary,
/// A is this released binary, started without `RUSTFS_S3_STACK`.
/// A is this released binary.
Reference(PathBuf),
}
+2 -2
View File
@@ -31,8 +31,8 @@ pub mod chaos;
#[cfg(test)]
pub mod fake_s3_target;
// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
// Differential (rustfs/backlog#2740): two processes, the binary under test against
// itself or a released reference binary, the same SDK script for every S3 operation,
// compared on status, error code, allowlisted headers and typed body fields.
#[cfg(test)]
mod dual_stack_differential;
+2 -2
View File
@@ -454,8 +454,8 @@ pub mod data_usage {
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_compression_total_from_memory,
load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
+1 -1
View File
@@ -1405,7 +1405,7 @@ impl BucketMetadata {
/// | public access block | Fails closed in `get_public_access_block_config`: degrading grants the anonymous access the operator asked to block. |
/// | quota | Fails closed in `get_quota_config`; the enforcement path in `quota::checker` already re-parses the raw JSON and refuses on error. |
/// | lifecycle | `get_lifecycle_config` fails closed, so GetBucketLifecycle reports the fault instead of NoSuchLifecycleConfiguration. ILM, scanner and expiry-header consumers still degrade to "no rules": nothing is deleted or moved on the strength of an unreadable rule set, and the bucket keeps serving reads and writes. |
/// | notification | `get_notification_config` fails closed, so GetBucketNotificationConfiguration reports the fault and startup leaves that one bucket's rules unchanged instead of clearing them; other buckets are unaffected. |
/// | notification | `get_notification_config` fails closed, so GetBucketNotificationConfiguration reports the fault and startup skips that bucket's rule registration; metadata-bound server dispatch suppresses external delivery for the unreadable snapshot. Other buckets are unaffected. |
/// | tagging, CORS, logging, website, accelerate, request payment | The getter fails closed so the matching GET API reports the fault instead of "not configured". Per-request consumers (CORS response headers) still degrade to "not configured", which is the restrictive direction. |
/// | bucket ACL | Safe to degrade: it only shapes an optional response. |
///
+62
View File
@@ -2612,6 +2612,10 @@ impl BucketMetadataSys {
}
}
pub(crate) async fn cached_metadata(&self) -> Vec<Arc<BucketMetadata>> {
self.metadata_map.read().await.values().cloned().collect()
}
pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
self.revoke_pending_bucket_creation(&bucket).await;
bump_local_meta_mutation_gen();
@@ -5760,6 +5764,64 @@ mod tests {
assert_eq!(tags.tag_set.len(), WRITERS, "every concurrent rewrite must survive: {tags:?}");
}
#[tokio::test]
async fn notification_snapshots_follow_periodic_refresh_after_lost_peer_signals() {
let (dirs, store) = isolated_store_over_temp_disks().await;
let writer = BucketMetadataSys::new(store.clone());
let peer = Arc::new(RwLock::new(BucketMetadataSys::new(store.clone())));
let bucket = "notification-periodic-snapshots";
for dir in &dirs {
std::fs::create_dir_all(dir.path().join(bucket)).expect("create empty fixture bucket");
}
let mut original = BucketMetadata::new(bucket);
original.notification_config_xml = concat!(
"<NotificationConfiguration><QueueConfiguration><Event>s3:ObjectCreated:Put</Event>",
"<Queue>arn:rustfs:sqs:us-east-1:inert:webhook</Queue></QueueConfiguration></NotificationConfiguration>"
)
.as_bytes()
.to_vec();
original.parse_all_configs().expect("parse inert subscription");
writer
.persist_new_and_set(original)
.await
.expect("persist initial subscription");
peer.read().await.reload_from_store(bucket).await.expect("prime peer cache");
let old = peer.read().await.get(bucket).await.expect("cached original snapshot");
let mut removed = (*old).clone();
removed.notification_config_xml = b"<NotificationConfiguration/>".to_vec();
removed.parse_all_configs().expect("parse removed subscription");
writer
.persist_new_and_set(removed)
.await
.expect("persist removal without peer callback");
assert!(Arc::ptr_eq(&old, &peer.read().await.get(bucket).await.expect("peer remains stale")));
// Exercise the timer's actual one-pass entry point, not the peer RPC.
refresh_buckets_metadata_once(peer.clone()).await;
let snapshots = peer.read().await.cached_metadata().await;
assert_eq!(snapshots.len(), 1);
assert!(!Arc::ptr_eq(&old, &snapshots[0]));
assert_eq!(old.bucket_incarnation_id, snapshots[0].bucket_incarnation_id);
assert_eq!(snapshots[0].notification_config_xml, b"<NotificationConfiguration/>");
assert!(
snapshots[0]
.notification_config
.as_ref()
.expect("parsed removal")
.behavior()
.queue_routes
.is_empty()
);
for dir in &dirs {
std::fs::remove_dir(dir.path().join(bucket)).expect("remove empty fixture bucket without peer callback");
}
refresh_buckets_metadata_once(peer.clone()).await;
assert!(peer.read().await.cached_metadata().await.is_empty());
assert!(peer.read().await.get(bucket).await.is_err());
}
/// Pins the peer reload-notification contract (`reload_from_store`, the
/// LoadBucketMetadata RPC path): only metadata actually read from
/// persisted storage enters the cache. A load miss errors out and leaves
+21 -116
View File
@@ -1463,36 +1463,22 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
let observed = load_observed_data_usage_snapshot(store.clone()).await.ok().flatten();
let (selected, selected_is_current_format) =
select_admin_data_usage_snapshot(authoritative, source.is_authoritative(), observed);
Ok(normalize_loaded_data_usage(selected, selected_is_current_format).await.0)
}
async fn project_admin_data_usage_for_current_buckets(
store: Arc<ECStore>,
mut response: DataUsageInfo,
with_memory_overlay: bool,
) -> DataUsageInfo {
let mut response = normalize_loaded_data_usage(selected, selected_is_current_format).await.0;
use futures::StreamExt as _;
let snapshot_measured = response.is_complete_bucket_usage_snapshot() || response.is_valid_partial_snapshot();
let overlay_authoritative = with_memory_overlay && snapshot_measured && !runtime_sources::setup_is_dist_erasure().await;
let deadline = tokio::time::Instant::now() + ADMIN_USAGE_INCARNATION_TIMEOUT;
let mut incarnations = HashMap::new();
{
// Identity reads happen after releasing snapshot and refresh locks,
// including on cache hits. Metadata transaction reads linearize each
// bucket against creation/deletion without relying on peer notices.
// The common deadline cancels unfinished work and bounds added latency.
let mut buckets = response
.buckets_usage
// Display-only identity reads must not serialize every bucket behind a
// slow metadata lock while holding the shared admin refresh permit.
// The common deadline bounds their total added latency; dropping this
// stream cancels all unfinished reads without spawning detached work.
// Own the names and store across awaits so this future remains Send
// when composed into spawned metrics collectors.
let buckets = response
.usage_snapshot_bucket_observations
.keys()
.chain(response.usage_snapshot_bucket_observations.keys())
.cloned()
.collect::<HashSet<_>>();
if overlay_authoritative {
// Copy only names while holding the memory lock, then release it
// before storage I/O. A later new entry has no identity proof and
// will remain unknown in this response.
buckets.extend(memory_cache().read().await.keys().cloned());
}
.collect::<Vec<_>>();
let mut identities = futures::stream::iter(buckets)
.map(move |bucket| {
let store = store.clone();
@@ -1506,6 +1492,7 @@ async fn project_admin_data_usage_for_current_buckets(
let Ok(Some((bucket, identity))) = tokio::time::timeout_at(deadline, identities.next()).await else {
break;
};
// Unavailable, timed-out or recreated buckets retain canonical values.
if let Ok(incarnation) = identity
&& !incarnation.is_nil()
{
@@ -1514,17 +1501,7 @@ async fn project_admin_data_usage_for_current_buckets(
}
}
response.project_bucket_observations_for_admin(&incarnations);
if apply_bucket_usage_memory_overlay_if_authoritative(&mut response, overlay_authoritative).await {
response.usage_snapshot_complete = false;
response.usage_snapshot_partial = true;
response.usage_snapshot_converged = Some(false);
response.usage_snapshot_admin_projection = true;
response.replication_info.clear();
response.tier_stats = None;
response.unknown_tier_stats = None;
}
response.retain_current_bucket_usage_for_admin(&incarnations);
response
Ok(response)
}
fn discard_incomplete_bucket_usage(data_usage_info: &mut DataUsageInfo) {
@@ -1713,7 +1690,11 @@ async fn project_data_usage_from_backend_cached<T>(
}
}
async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
/// Load the freshest structurally complete snapshot for authenticated admin
/// observability. A scan raced by namespace activity may be selected here, but
/// never by [`load_data_usage_from_backend_cached`], which remains the
/// converged source for quota admission.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
loop {
@@ -1762,38 +1743,6 @@ async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<Da
}
}
/// Load cached scanner statistics for authenticated admin observability.
///
/// Each response verifies bucket identities against current storage metadata,
/// including cache hits. Missing, failed, or timed-out identity reads leave that
/// bucket's usage absent (unknown). Legacy values without an incarnation stay
/// unknown until a scanner snapshot certifies their identity. Per-bucket reads
/// linearize independently; this is not an atomic namespace snapshot.
/// Identity reads have a shared deadline and do not outlive cancellation. The
/// unchanged cached baseline remains independent of quota and persistence.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, false).await)
}
/// Load the admin view with the existing authoritative single-process overlay.
///
/// Only a complete or valid partial scanner snapshot can receive memory values.
/// The final identity check excludes statistics from retired or unverified buckets.
/// Distributed nodes continue to use scanner statistics without local totals.
/// Cancellation, unknown values, and per-bucket read semantics are the same as
/// [`load_admin_data_usage_from_backend_cached`]. No overlay is persisted.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_with_memory_overlay(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, true).await)
}
/// Invalidate the process-local persisted usage snapshot cache after a durable
/// scanner write.
pub async fn invalidate_data_usage_snapshot_cache() {
@@ -2856,17 +2805,16 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
*cache = next_cache;
}
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) -> bool {
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) {
if !authoritative {
return false;
return;
}
let cache = memory_cache().read().await;
if cache.is_empty() {
return false;
return;
}
let was_complete = data_usage_info.usage_snapshot_complete;
let persisted_update = data_usage_info.last_update;
let mut changed = false;
let mut added_bucket = false;
@@ -2903,7 +2851,6 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
data_usage_info.calculate_totals();
}
changed || was_complete != data_usage_info.usage_snapshot_complete
}
pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
@@ -3797,23 +3744,6 @@ mod tests {
*cache_updating().write().await = false;
}
async fn bind_admin_usage_test_identities(store: &Arc<ECStore>, snapshot: &mut DataUsageInfo) {
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
for (bucket, usage) in &mut snapshot.buckets_usage {
store
.make_bucket(bucket, &crate::storage_api_contracts::bucket::MakeBucketOptions::default())
.await
.expect("create a real bucket for current admin identity validation");
usage.bucket_incarnation = Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("read the storage-owned fixture identity")
.to_string(),
);
}
}
fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
let mut info = DataUsageInfo {
last_update: Some(last_update),
@@ -4349,11 +4279,7 @@ mod tests {
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let (mut authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
bind_admin_usage_test_identities(&store, &mut authoritative).await;
for (bucket, usage) in &mut observed.buckets_usage {
usage.bucket_incarnation = authoritative.buckets_usage[bucket].bucket_incarnation.clone();
}
let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
@@ -4402,11 +4328,6 @@ mod tests {
reconciled.scanner_epoch = Some(4);
reconciled.scanner_cycle = Some(12);
reconciled.usage_snapshot_converged = Some(true);
reconciled
.buckets_usage
.get_mut("large-bucket")
.expect("reconciled bucket")
.bucket_incarnation = authoritative.buckets_usage["large-bucket"].bucket_incarnation.clone();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4928,12 +4849,6 @@ mod tests {
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut current).await;
backup
.buckets_usage
.get_mut("existing")
.expect("backup bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
@@ -4956,11 +4871,6 @@ mod tests {
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
after_delete
.buckets_usage
.get_mut("existing")
.expect("post-delete bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
@@ -5022,7 +4932,6 @@ mod tests {
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut authoritative).await;
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -5037,10 +4946,6 @@ mod tests {
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info.buckets_usage
.get_mut("bucket")
.expect("observed bucket")
.bucket_incarnation = authoritative.buckets_usage["bucket"].bucket_incarnation.clone();
info
};
let first = observation(7, 11, 200, 11, 1_100);
@@ -221,7 +221,7 @@ async fn companion_admin_cache_isolated_from_quota_memory_and_persistence() {
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_buckets() {
async fn blocked_companion_identity_keeps_canonical_without_delaying_healthy_buckets() {
use crate::bucket::metadata_sys::{
acquire_metadata_transaction_guard, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
@@ -280,11 +280,8 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
.await
.expect("optional projection must return before the blocked metadata lock's long timeout")
.expect("canonical admin response stays available");
assert!(
!response.buckets_usage.contains_key(blocked),
"an unavailable current identity cannot bind the canonical value"
);
assert!(!response.bucket_sizes.contains_key(blocked));
assert_eq!(response.buckets_usage[blocked].size, 100);
assert_eq!(response.buckets_usage[blocked].usage_value_state, Some(BucketUsageValueState::Carried));
assert_eq!(response.buckets_usage[healthy].size, 250);
assert_eq!(response.buckets_usage[healthy].usage_value_state, Some(BucketUsageValueState::Observed));
assert!(response.usage_snapshot_admin_projection);
@@ -296,12 +293,13 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
.is_ok()
);
drop(blocked_guard);
invalidate_data_usage_snapshot_cache().await;
let recovered = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh after lock recovery");
assert_eq!(
recovered.buckets_usage[blocked].size, 250,
"unavailable identity can recover on a cache hit"
"unavailable identity can recover on the next refresh"
);
assert_eq!(recovered.buckets_usage[healthy].size, 250);
let quota = load_data_usage_from_backend_cached(store)
@@ -310,249 +308,3 @@ async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_bucke
assert_eq!(quota.buckets_usage[blocked].size, 100);
assert_eq!(quota.buckets_usage[healthy].size, 100);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn cached_canonical_admin_usage_rechecks_current_identity_without_peer_invalidation() {
use crate::bucket::metadata::{load_bucket_incarnation, save_bucket_incarnation};
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-cache";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
canonical.usage_snapshot_pending_buckets.clear();
canonical.usage_snapshot_pending_bucket_count = 0;
canonical
.buckets_usage
.get_mut(bucket)
.expect("canonical bucket")
.objects_count = 107;
canonical.calculate_totals();
let original_bytes = serde_json::to_vec(&canonical).expect("encode canonical old bucket");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), original_bytes.clone())
.await
.expect("persist canonical root");
invalidate_data_usage_snapshot_cache().await;
let first = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("prime raw admin cache");
assert_eq!(first.buckets_usage[bucket].objects_count, 107);
assert!(first.is_complete_bucket_usage_snapshot());
let cache_generation = admin_data_usage_snapshot_generation();
let new_identity = uuid::Uuid::new_v4();
// Simulate the storage-owned identity changing on another node while this
// process has not received its invalidation notice. Do not clear the cache.
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish changed disk identity");
assert_eq!(
load_bucket_incarnation(store.clone(), bucket)
.await
.expect("read new sidecar"),
Some(new_identity)
);
let second = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("read raw cache with fresh identity");
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
assert!(
!second.buckets_usage.contains_key(bucket),
"same name cannot retain the predecessor's 107"
);
assert!(!second.bucket_sizes.contains_key(bucket));
assert_eq!(second.usage_snapshot_converged, Some(false));
assert!(second.usage_snapshot_admin_projection);
assert!(reject_admin_usage_projection(&second).is_err());
assert!(certified_bucket_usage(&second, bucket, new_identity).is_none());
assert!(parse_usage_snapshot(&serde_json::to_vec(&second).expect("encode display response")).is_err());
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("independent quota baseline");
assert_eq!(quota.buckets_usage[bucket].objects_count, 107);
assert_eq!(
read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("unchanged persisted root"),
original_bytes
);
// Rebinding a cached raw baseline is transient: recovery does not need to
// reload or permanently blank a healthy bucket's statistics.
save_bucket_incarnation(store.clone(), bucket, old_identity)
.await
.expect("restore fixture identity");
let recovered = load_admin_data_usage_from_backend_cached(store)
.await
.expect("recover on another cache hit");
assert_eq!(recovered.buckets_usage[bucket].objects_count, 107);
assert!(recovered.is_complete_bucket_usage_snapshot());
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_requires_a_measured_scanner_baseline() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(!runtime_sources::setup_is_dist_erasure().await);
let bucket = "usage-unmeasured-overlay";
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap();
let mut baseline = data_usage_info_for_test(bucket, 1, 100, SystemTime::UNIX_EPOCH);
let usage = baseline.buckets_usage.get_mut(bucket).unwrap();
usage.bucket_incarnation = Some(incarnation.to_string());
let memory_usage = BucketUsageInfo {
size: 150,
..usage.clone()
};
clear_usage_memory_cache_for_test().await;
replace_bucket_usage_memory_from_authoritative(bucket, memory_usage, SystemTime::now()).await;
// A missing snapshot and a mismatched bucket count both leave usage unknown.
for (buckets_count, expected_size) in [(None, None), (Some(2), None), (Some(1), Some(150))] {
if let Some(count) = buckets_count {
baseline.buckets_count = count;
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), serde_json::to_vec(&baseline).unwrap())
.await
.unwrap();
}
invalidate_data_usage_snapshot_cache().await;
let response = load_admin_data_usage_with_memory_overlay(store.clone()).await.unwrap();
assert_eq!(
response.buckets_usage.get(bucket).map(|usage| usage.size),
expected_size,
"baseline count {buckets_count:?}"
);
assert_eq!(response.bucket_sizes.get(bucket).copied(), expected_size);
assert_eq!(response.objects_total_size, expected_size.unwrap_or_default());
assert_eq!(response.usage_snapshot_admin_projection, expected_size.is_some());
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
}
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_cannot_reintroduce_a_predecessor_identity() {
use crate::bucket::metadata::save_bucket_incarnation;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-overlay";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode canonical root"),
)
.await
.expect("persist overlay fixture");
invalidate_data_usage_snapshot_cache().await;
let cached_usage = canonical.buckets_usage[bucket].clone();
replace_bucket_usage_memory_from_authoritative(bucket, cached_usage, SystemTime::now()).await;
let new_identity = uuid::Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish a replacement identity");
let response = load_admin_data_usage_with_memory_overlay(store.clone())
.await
.expect("project after memory overlay");
assert!(!response.buckets_usage.contains_key(bucket));
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(response.usage_snapshot_admin_projection);
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_overlay_flags_only_change_keeps_identity_bound_cold_usage() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(
!runtime_sources::setup_is_dist_erasure().await,
"exercise the production single-process overlay"
);
let cold = "usage-overlay-cold";
let unverified = "usage-overlay-unverified";
for bucket in [cold, unverified] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay fixture bucket");
}
let identity = store
.bucket_incarnation_id_from_disk(cold)
.await
.expect("cold bucket identity");
let mut canonical = companion_snapshot(cold, identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode cold canonical value"),
)
.await
.expect("persist cold baseline");
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
memory_cache().write().await.insert(
unverified.to_string(),
cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false),
);
record_bucket_object_write_memory(unverified, None, 42).await;
{
let cache = memory_cache().read().await;
assert!(cache[unverified].dirty);
assert!(!cache[unverified].authoritative);
}
let response = load_admin_data_usage_with_memory_overlay(store)
.await
.expect("identity-filtered flags-only overlay");
assert_eq!(response.buckets_usage[cold].size, 100);
assert_eq!(response.objects_total_size, 100);
assert_eq!(response.bucket_sizes[cold], 100);
assert!(!response.buckets_usage.contains_key(unverified));
assert!(response.usage_snapshot_admin_projection);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
@@ -1033,26 +1033,16 @@ mod tests {
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket("bucket", &MakeBucketOptions::default())
.await
.expect("create a real observed bucket");
let incarnation = store
.bucket_incarnation_id_from_disk("bucket")
.await
.expect("read current storage-owned incarnation");
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
@@ -1076,7 +1066,6 @@ mod tests {
versions_count: 4,
delete_markers_count: 1,
size: 6,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
},
)]
@@ -1128,46 +1117,6 @@ mod tests {
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn identity_filtered_usage_remains_unknown_in_server_info() {
let mut info = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
buckets_count: 1,
objects_total_count: 107,
objects_total_size: 1024,
buckets_usage: [(
"bucket".to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 107,
size: 1024,
bucket_incarnation: Some("11111111-1111-4111-8111-111111111111".to_string()),
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
..Default::default()
};
let current =
std::collections::HashMap::from([("bucket".to_string(), "22222222-2222-4222-8222-222222222222".to_string())]);
assert_eq!(info.retain_current_bucket_usage_for_admin(&current), 1);
assert!(info.buckets_usage.is_empty());
assert!(info.usage_snapshot_admin_projection);
assert_eq!(info.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
+9
View File
@@ -775,6 +775,15 @@ impl ECStore {
sys.read().await.get(bucket).await
}
/// Snapshots this instance's cached bucket metadata without disk IO.
///
/// # Errors
/// Returns an error when this instance's metadata system is not initialized.
pub async fn cached_bucket_metadata(&self) -> Result<Vec<Arc<BucketMetadata>>> {
let sys = metadata_sys::require_bucket_metadata_sys_in(&self.ctx)?;
Ok(sys.read().await.cached_metadata().await)
}
/// Resolve stored tags for authorization. Only confirmed absence returns an
/// empty map; unavailable, fabricated or malformed metadata must not grant access.
pub async fn get_bucket_tags_for_policy(&self, bucket: &str) -> Result<HashMap<String, String>> {
+7 -1
View File
@@ -47,12 +47,18 @@ impl NotifyBucketConfigManager {
}
pub async fn has_subscriber(&self, bucket: &str, event: &EventName) -> bool {
if !self.subscriber_view.has_subscriber(bucket, event) {
// The standalone index is only a fast negative check for name-only
// rules. Metadata-bound rules can change without an index callback.
if !self.rule_engine.uses_bucket_metadata_source() && !self.subscriber_view.has_subscriber(bucket, event) {
return false;
}
self.rule_engine.has_subscriber(bucket, event).await
}
pub(crate) fn bind_bucket_metadata_source(&self, store: &Arc<crate::NotifyStore>) {
self.rule_engine.bind_bucket_metadata_source(store);
}
/// Validates that the runtime can accept `cfg` for `bucket` without
/// mutating any state.
pub async fn validate_bucket_notification_config(
+10
View File
@@ -249,6 +249,16 @@ impl NotificationSystem {
self.services.bucket_config_manager.has_subscriber(bucket, event).await
}
/// Binds server dispatch to the store-owned metadata cache. Local writes,
/// peer reloads, deletes and periodic metadata refreshes then select rules
/// from the same snapshot; delayed name-only rule replacements are ignored
/// and clears only evict compiled snapshots. Missing/unreadable metadata or
/// a dropped store suppresses external dispatch without falling back to the
/// standalone rules. Already queued or in-flight deliveries are not recalled.
pub fn bind_bucket_metadata_source(&self, store: &Arc<crate::NotifyStore>) {
self.services.bucket_config_manager.bind_bucket_metadata_source(store);
}
/// Returns true when at least one in-process consumer is subscribed to live events.
pub fn has_live_listeners(&self) -> bool {
self.services.pipeline.has_live_listeners()
+292 -6
View File
@@ -12,12 +12,14 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::rules::xml_config::{FilterRule, S3KeyFilter};
use crate::rules::{RulesMap, TargetIdSet};
use crate::storage_api::bucket_metadata::{BucketMetadata, NotifyStore};
use percent_encoding::percent_decode_str;
use rustfs_s3_types::EventName;
use rustfs_targets::arn::TargetID;
use rustfs_targets::arn::{ARN, TargetID};
use starshard::{AsyncShardedHashMap, DEFAULT_SHARDS, SnapshotMode};
use std::sync::Arc;
use std::sync::{Arc, RwLock, Weak};
use tracing::info;
const LOG_COMPONENT_NOTIFY: &str = "notify";
@@ -35,20 +37,82 @@ fn decoded_object_key_for_matching(object_key: &str) -> Option<String> {
#[derive(Clone)]
pub struct NotifyRuleEngine {
bucket_rules_map: Arc<AsyncShardedHashMap<String, RulesMap, rustc_hash::FxBuildHasher>>,
bucket_rules_map: Arc<AsyncShardedHashMap<String, CachedBucketRules, rustc_hash::FxBuildHasher>>,
// Never hold this short synchronous guard across an await. Metadata cache
// reads precede the compiled-rules cache, so there is no reverse lock order.
metadata_source: Arc<RwLock<Option<Weak<NotifyStore>>>>,
}
#[derive(Clone)]
struct CachedBucketRules {
metadata: Option<Weak<BucketMetadata>>,
// Cache rejected snapshots too, so corrupt persisted routes are not
// repeatedly compiled on the event hot path.
rules: Option<RulesMap>,
}
fn rules_from_metadata(metadata: &BucketMetadata) -> Option<RulesMap> {
let mut rules = RulesMap::new();
let Some(config) = metadata.notification_config.as_ref() else {
// Unreadable metadata is not authority to keep an old subscription.
return metadata.notification_config_xml.is_empty().then_some(rules);
};
let config = config.behavior();
for route in config
.queue_routes
.into_iter()
.chain(config.topic_routes)
.chain(config.lambda_routes)
{
let target = ARN::parse(&route.destination).ok()?.target_id;
let events: Vec<_> = route.events.iter().filter_map(|event| EventName::parse(event).ok()).collect();
let mut filter = S3KeyFilter::default();
for (name, value) in route.filter_rules.into_iter().flatten() {
filter.filter_rule_list.push(FilterRule {
name: name?,
value: value?,
});
}
filter.validate().ok()?;
rules.add_rule_config(&events, filter.pattern(), target);
}
Some(rules)
}
impl NotifyRuleEngine {
pub fn new() -> Self {
Self {
bucket_rules_map: Arc::new(AsyncShardedHashMap::with_snapshot_mode(DEFAULT_SHARDS, SnapshotMode::Cached)),
metadata_source: Arc::new(RwLock::new(None)),
}
}
/// Uses this store's immutable metadata snapshots as rule authority. Once
/// bound, missing metadata or a dropped store fails closed, never falling
/// back to the standalone name-only table. The weak handle avoids cycles.
pub fn bind_bucket_metadata_source(&self, store: &Arc<NotifyStore>) {
*self.metadata_source.write().unwrap_or_else(|err| err.into_inner()) = Some(Arc::downgrade(store));
}
fn metadata_source(&self) -> Option<Weak<NotifyStore>> {
self.metadata_source.read().unwrap_or_else(|err| err.into_inner()).clone()
}
pub(crate) fn uses_bucket_metadata_source(&self) -> bool {
self.metadata_source().is_some()
}
pub async fn is_target_bound_to_any_bucket(&self, target_id: &TargetID) -> bool {
if let Some(source) = self.metadata_source() {
let Some(store) = source.upgrade() else { return true };
let Ok(metadata) = store.cached_bucket_metadata().await else { return true };
return metadata
.iter()
.any(|metadata| rules_from_metadata(metadata).is_none_or(|rules| rules.contains_target_id(target_id)));
}
let items = self.bucket_rules_map.iter().await;
for (_bucket, rules_map) in items {
if rules_map.contains_target_id(target_id) {
if rules_map.rules.is_some_and(|rules| rules.contains_target_id(target_id)) {
return true;
}
}
@@ -56,11 +120,24 @@ impl NotifyRuleEngine {
}
pub async fn set_bucket_rules(&self, bucket: &str, rules_map: RulesMap) {
if self.uses_bucket_metadata_source() {
// Delayed PUT/startup callbacks cannot override a newer revision or
// a same-name replacement. Only published metadata is authoritative.
return;
}
let event_count = rules_map.iter_events().count();
if rules_map.is_empty() {
self.bucket_rules_map.remove(&bucket.to_string()).await;
} else {
self.bucket_rules_map.insert(bucket.to_string(), rules_map).await;
self.bucket_rules_map
.insert(
bucket.to_string(),
CachedBucketRules {
metadata: None,
rules: Some(rules_map),
},
)
.await;
}
info!(
event = EVENT_NOTIFY_RULES_UPDATED,
@@ -74,7 +151,47 @@ impl NotifyRuleEngine {
}
pub async fn get_bucket_rules(&self, bucket: &str) -> Option<RulesMap> {
self.bucket_rules_map.get(&bucket.to_string()).await
if let Some(source) = self.metadata_source() {
let metadata = match source.upgrade() {
Some(store) => store.get_bucket_metadata(bucket).await.ok(),
None => None,
};
return self.rules_for_metadata(bucket, metadata).await;
}
self.bucket_rules_map
.get(&bucket.to_string())
.await
.and_then(|cached| cached.rules)
}
async fn rules_for_metadata(&self, bucket: &str, metadata: Option<Arc<BucketMetadata>>) -> Option<RulesMap> {
let key = bucket.to_owned();
let Some(metadata) = metadata else {
self.bucket_rules_map.remove(&key).await;
return None;
};
if let Some(cached) = self.bucket_rules_map.get(&key).await
&& cached
.metadata
.as_ref()
.is_some_and(|snapshot| Weak::ptr_eq(snapshot, &Arc::downgrade(&metadata)))
{
return cached.rules;
}
// Snapshot identity covers incarnation, revision AND bytes, without
// trusting wall-clock ordering. A late compilation may evict a newer
// cache entry, but cannot be reused by a reader of a different snapshot.
let rules = rules_from_metadata(&metadata);
self.bucket_rules_map
.insert(
key,
CachedBucketRules {
metadata: Some(Arc::downgrade(&metadata)),
rules: rules.clone(),
},
)
.await;
rules
}
pub async fn clear_bucket_rules(&self, bucket: &str) {
@@ -119,8 +236,177 @@ impl Default for NotifyRuleEngine {
mod tests {
use super::NotifyRuleEngine;
use crate::rules::RulesMap;
use crate::storage_api::bucket_metadata::{BucketMetadata, deserialize_bucket_config};
use rustfs_s3_types::EventName;
use rustfs_targets::arn::TargetID;
use std::sync::{Arc, Weak};
fn metadata(target: &str) -> Arc<BucketMetadata> {
let mut metadata = BucketMetadata::new("bucket");
metadata.notification_config_xml = format!(
"<NotificationConfiguration><QueueConfiguration><Event>s3:ObjectCreated:Put</Event>\
<Queue>arn:rustfs:sqs:us-east-1:{target}:webhook</Queue></QueueConfiguration></NotificationConfiguration>"
)
.into_bytes();
metadata.notification_config =
Some(deserialize_bucket_config(&metadata.notification_config_xml).expect("parse inert notification fixture"));
Arc::new(metadata)
}
#[tokio::test]
async fn notification_snapshot_cache_never_reuses_replaced_incarnation_or_revision() {
let engine = NotifyRuleEngine::new();
let old = metadata("old");
let mut replacement = (*metadata("new")).clone();
// Identical timestamps must not make different configurations equal.
replacement.notification_config_updated_at = old.notification_config_updated_at;
let replacement = Arc::new(replacement);
assert_ne!(old.bucket_incarnation_id, replacement.bucket_incarnation_id);
let old_target = TargetID::new("old".into(), "webhook".into());
let new_target = TargetID::new("new".into(), "webhook".into());
for (snapshot, wanted, unwanted) in [
(old.clone(), &old_target, &new_target),
(replacement.clone(), &new_target, &old_target),
// Simulate a late compilation of a captured, older snapshot.
(old.clone(), &old_target, &new_target),
(replacement.clone(), &new_target, &old_target),
] {
let rules = engine
.rules_for_metadata("bucket", Some(snapshot))
.await
.expect("readable snapshot");
let targets = rules.match_rules(EventName::ObjectCreatedPut, "object");
assert!(targets.contains(wanted));
assert!(!targets.contains(unwanted));
}
let mut next_revision = (*old).clone();
next_revision.notification_config = replacement.notification_config.clone();
next_revision.notification_config_xml = replacement.notification_config_xml.clone();
let rules = engine
.rules_for_metadata("bucket", Some(Arc::new(next_revision)))
.await
.expect("same-incarnation revision");
assert!(!rules.contains_target_id(&old_target));
assert!(rules.contains_target_id(&new_target));
}
#[tokio::test]
async fn notification_snapshot_absence_and_corruption_never_fall_back_to_old_rules() {
let engine = NotifyRuleEngine::new();
let old = metadata("old");
assert!(engine.rules_for_metadata("bucket", Some(old.clone())).await.is_some());
let mut unreadable = (*old).clone();
unreadable.notification_config = None;
unreadable.notification_config_xml = b"<broken>".to_vec();
assert!(
engine
.rules_for_metadata("bucket", Some(Arc::new(unreadable)))
.await
.is_none()
);
assert!(
engine
.bucket_rules_map
.get(&"bucket".to_owned())
.await
.expect("cache rejected snapshot")
.rules
.is_none()
);
assert!(engine.rules_for_metadata("bucket", None).await.is_none());
assert!(engine.bucket_rules_map.get(&"bucket".to_owned()).await.is_none());
// A recreated bucket without notifications has no inherited targets.
let replacement = Arc::new(BucketMetadata::new("bucket"));
assert!(
engine
.rules_for_metadata("bucket", Some(replacement))
.await
.expect("empty snapshot")
.is_empty()
);
}
#[tokio::test]
async fn notification_snapshot_binding_never_falls_back_after_store_loss() {
let engine = NotifyRuleEngine::new();
let target = TargetID::new("old".into(), "webhook".into());
let mut rules = RulesMap::new();
rules.add_rule_config(&[EventName::ObjectCreatedPut], "*".into(), target.clone());
engine.set_bucket_rules("bucket", rules.clone()).await;
assert!(engine.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
*engine.metadata_source.write().expect("metadata source lock") = Some(Weak::new());
// A delayed validated PUT must not revive the standalone table, even
// when its owning store is gone. Target removal also fails closed.
engine.set_bucket_rules("bucket", rules).await;
assert!(!engine.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
assert!(
engine
.match_targets("bucket", EventName::ObjectCreatedPut, "object")
.await
.is_empty()
);
assert!(engine.is_target_bound_to_any_bucket(&target).await);
}
#[test]
fn notification_snapshot_compilation_preserves_destinations_events_and_filters() {
let mut metadata = (*metadata("queue")).clone();
metadata.notification_config_xml = concat!(
"<NotificationConfiguration>",
"<QueueConfiguration><Event>s3:ObjectCreated:*</Event><Event>s3:Future:Unrecognized</Event>",
"<Queue>arn:rustfs:sqs:us-east-1:queue:webhook</Queue>",
"<Filter><S3Key><FilterRule><Name>Prefix</Name><Value>logs/</Value></FilterRule>",
"<FilterRule><Name>Suffix</Name><Value>.csv</Value></FilterRule></S3Key></Filter></QueueConfiguration>",
"<TopicConfiguration><Event>s3:ObjectRemoved:*</Event><Topic>arn:rustfs:sqs:us-east-1:topic:webhook</Topic></TopicConfiguration>",
"<CloudFunctionConfiguration><Event>s3:ObjectCreated:Put</Event>",
"<CloudFunction>arn:rustfs:sqs:us-east-1:lambda:webhook</CloudFunction></CloudFunctionConfiguration>",
"</NotificationConfiguration>"
).as_bytes().to_vec();
metadata.notification_config =
Some(deserialize_bucket_config(&metadata.notification_config_xml).expect("parse destinations"));
let rules = super::rules_from_metadata(&metadata).expect("compile destinations");
assert_eq!(rules.match_rules(EventName::ObjectCreatedPut, "logs/data.csv").len(), 2);
assert_eq!(rules.match_rules(EventName::ObjectCreatedPut, "other/data.csv").len(), 1);
assert_eq!(rules.match_rules(EventName::ObjectCreatedPut, "logs/data.txt").len(), 1);
let removed = rules.match_rules(EventName::ObjectRemovedDelete, "object");
assert_eq!(removed.len(), 1);
assert!(removed.contains(&TargetID::new("topic".into(), "webhook".into())));
}
#[test]
fn notification_snapshot_compilation_rejects_invalid_persisted_routes() {
for (destination, filter) in [
("not-an-arn", ""),
(
"arn:rustfs:sqs:us-east-1:inert:webhook",
"<Filter><S3Key><FilterRule><Value>logs/</Value></FilterRule></S3Key></Filter>",
),
(
"arn:rustfs:sqs:us-east-1:inert:webhook",
concat!(
"<Filter><S3Key><FilterRule><Name>prefix</Name><Value>logs/</Value></FilterRule>",
"<FilterRule><Name>Prefix</Name><Value>private/</Value></FilterRule></S3Key></Filter>"
),
),
] {
let mut metadata = BucketMetadata::new("bucket");
metadata.notification_config_xml = format!(
"<NotificationConfiguration><QueueConfiguration><Event>s3:ObjectCreated:Put</Event>\
<Queue>{destination}</Queue>{filter}</QueueConfiguration></NotificationConfiguration>"
)
.into_bytes();
metadata.notification_config = Some(
deserialize_bucket_config(&metadata.notification_config_xml)
.expect("parse invalid but well-formed persisted route"),
);
assert!(super::rules_from_metadata(&metadata).is_none(), "invalid route must not dispatch");
}
}
#[tokio::test]
async fn rule_engine_tracks_bucket_rule_lifecycle() {
+7
View File
@@ -70,3 +70,10 @@ pub(crate) mod crate_boundary {
save_notify_server_config_snapshot, with_notify_server_config_read_lock,
};
}
pub(crate) mod bucket_metadata {
pub(crate) use super::NotifyStore;
pub(crate) use rustfs_ecstore::api::bucket::metadata::BucketMetadata;
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::bucket::utils::deserialize as deserialize_bucket_config;
}
-1
View File
@@ -15,7 +15,6 @@
pub mod init_time;
pub mod last_minute;
pub mod metrics;
pub mod scan_observation;
pub mod startup;
pub mod usage_candidate;
+1 -40
View File
@@ -13,9 +13,6 @@
// limitations under the License.
use crate::last_minute::{AccElem, LastMinuteLatency};
use crate::scan_observation::{
ScannerMetadataReadGuard, ScannerMetadataReadOutcome, ScannerScanObservation, ScannerScanObservationSnapshot,
};
use crate::usage_candidate::{
ScannerUsageCandidateCount, ScannerUsageCandidateKind, ScannerUsageCandidateOutcome, USAGE_CANDIDATE_COUNTER_SLOTS,
};
@@ -980,7 +977,6 @@ impl ScannerUsagePublicationObservation {
pub struct Metrics {
startup: crate::startup::ScannerStartupObservation,
scan_observation: ScannerScanObservation,
operations: Vec<AtomicU64>,
latency: Vec<LockedLastMinuteLatency>,
actions: Vec<AtomicU64>,
@@ -1635,9 +1631,6 @@ pub struct ScannerMetricsReport {
pub lifecycle_transition: ScannerLifecycleTransitionSnapshot,
#[serde(default)]
pub usage_freshness: ScannerUsageFreshnessSnapshot,
/// Node-local, process-lifetime work observations. Older nodes have no evidence.
#[serde(default)]
pub scan_observation: ScannerScanObservationSnapshot,
/// Process-lifetime candidate-construction results. Missing on older reports
/// means unobserved, not accepted or healthy. Publication is tracked separately.
#[serde(default)]
@@ -2132,38 +2125,9 @@ impl Metrics {
&self.startup
}
pub fn scan_observation(&self) -> &ScannerScanObservation {
&self.scan_observation
}
/// Start after scanner node admission, before polling the disk read future.
/// Disk-permit and RPC waits remain included; decoding is excluded. Dropping
/// an unfinished guard observes cancellation, not physical I/O completion.
pub fn start_scanner_metadata_read(&self) -> ScannerMetadataReadGuard<'_> {
self.scan_observation.start_metadata_read();
ScannerMetadataReadGuard {
metrics: self,
started: std::time::Instant::now(),
outcome: ScannerMetadataReadOutcome::Cancelled,
bytes: 0,
}
}
pub(crate) fn finish_scanner_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
let bytes = if outcome == ScannerMetadataReadOutcome::Success {
bytes
} else {
0
};
self.scan_observation.finish_metadata_read(outcome, bytes, elapsed);
self.operations[Metric::ReadMetadata as usize].fetch_add(1, Ordering::Relaxed);
self.latency[Metric::ReadMetadata as usize].add_size(elapsed, bytes);
}
pub fn new() -> Self {
Self {
startup: crate::startup::ScannerStartupObservation::default(),
scan_observation: ScannerScanObservation::default(),
operations: (0..Metric::Last as usize).map(|_| AtomicU64::new(0)).collect(),
// Each slot gets its own fresh LockedLastMinuteLatency so that
// different metrics never accidentally share state.
@@ -2769,8 +2733,6 @@ impl Metrics {
/// walked the whole namespace, so an operator can tell why scoped
/// convergence is not happening.
pub fn record_scanner_scope_fallback(&self, reason: impl Into<String>) {
let reason = reason.into();
self.scan_observation.record_scope_fallback(&reason);
self.scanner_scope_fallback_total.fetch_add(1, Ordering::Relaxed);
self.scanner_scope_fallback_last_unix_secs
.store(unix_now_secs(), Ordering::Relaxed);
@@ -2778,7 +2740,7 @@ impl Metrics {
Ok(guard) => guard,
Err(poisoned) => poisoned.into_inner(),
};
*last_reason = reason;
*last_reason = reason.into();
}
pub fn record_scanner_usage_durable_success(&self) {
@@ -3913,7 +3875,6 @@ impl Metrics {
failed: self.scanner_transition_failed.load(Ordering::Relaxed),
};
m.usage_candidates = self.usage_candidate_counts();
m.scan_observation = self.scan_observation.snapshot();
let usage_save_result = ScannerUsageSaveResult::from_code(self.scanner_usage_last_save_result.load(Ordering::Relaxed));
m.usage_freshness = ScannerUsageFreshnessSnapshot {
dirty_pending_buckets: self.scanner_dirty_usage_pending_buckets.load(Ordering::Relaxed),
@@ -1,675 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Node-local, process-lifetime observations of scanner work. Counters are
//! independently loaded, so concurrent reports need not balance exactly.
//! These observations never grant cache reuse or publication authority.
use crate::metrics::Metrics;
use serde::{Deserialize, Serialize};
use std::{
collections::BTreeMap,
sync::atomic::{AtomicU64, Ordering},
time::{Duration, Instant},
};
pub const PREFIX_REUSE_BLOCKERS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
const SCOPE_FALLBACK_REASONS: [&str; 6] = [
"dirty_snapshot_incomplete",
"dirty_generation_exhausted",
"dirty_bucket_overflow",
"no_listed_dirty_buckets",
"baseline_unavailable",
"peer_scope_unavailable",
];
/// A storage read result, independent of later metadata decoding or ILM work.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerMetadataReadOutcome {
Success,
NotFound,
Error,
Cancelled,
}
/// The existing path that actually copied a cached subtree instead of walking it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseKind {
CleanPrefix,
CoveredNamespace,
SampledNamespace,
ClassifiedNamespace,
}
impl ScannerCacheReuseKind {
const ALL: [Self; 4] = [
Self::CleanPrefix,
Self::CoveredNamespace,
Self::SampledNamespace,
Self::ClassifiedNamespace,
];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::SampledNamespace => "sampled_namespace",
Self::ClassifiedNamespace => "classified_namespace",
}
}
}
/// Completed bucket-level reuse work, independent of durable publication.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBucketReuseKind {
ScopedColdBucket,
CurrentRoot,
}
impl ScannerBucketReuseKind {
const ALL: [Self; 2] = [Self::ScopedColdBucket, Self::CurrentRoot];
fn label(self) -> &'static str {
match self {
Self::ScopedColdBucket => "scoped_cold_bucket",
Self::CurrentRoot => "current_root",
}
}
}
/// A production folder gate, not a diagnostic preflight.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseGate {
CleanPrefix,
CoveredNamespace,
NamespaceSampling,
}
impl ScannerCacheReuseGate {
const ALL: [Self; 3] = [Self::CleanPrefix, Self::CoveredNamespace, Self::NamespaceSampling];
fn label(self) -> &'static str {
match self {
Self::CleanPrefix => "clean_prefix",
Self::CoveredNamespace => "covered_namespace",
Self::NamespaceSampling => "namespace_sampling",
}
}
}
/// The first existing condition reached by a folder reuse gate.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerCacheReuseDecision {
MissingScope,
NoResumeSweep,
DeepScan,
ForwardSweep,
ActiveReplication,
NotRoot,
CompactedParent,
MissingCachedChild,
InvalidPrefix,
DirtyPrefix,
ErasureHealthRequired,
MissingCoverage,
CachedFailures,
UncompactedChild,
SelectedForWalk,
Eligible,
}
impl ScannerCacheReuseDecision {
const ALL: [Self; 16] = [
Self::MissingScope,
Self::NoResumeSweep,
Self::DeepScan,
Self::ForwardSweep,
Self::ActiveReplication,
Self::NotRoot,
Self::CompactedParent,
Self::MissingCachedChild,
Self::InvalidPrefix,
Self::DirtyPrefix,
Self::ErasureHealthRequired,
Self::MissingCoverage,
Self::CachedFailures,
Self::UncompactedChild,
Self::SelectedForWalk,
Self::Eligible,
];
fn label(self) -> &'static str {
match self {
Self::MissingScope => "missing_scope",
Self::NoResumeSweep => "no_resume_sweep",
Self::DeepScan => "deep_scan",
Self::ForwardSweep => "forward_sweep",
Self::ActiveReplication => "active_replication",
Self::NotRoot => "not_root",
Self::CompactedParent => "compacted_parent",
Self::MissingCachedChild => "missing_cached_child",
Self::InvalidPrefix => "invalid_prefix",
Self::DirtyPrefix => "dirty_prefix",
Self::ErasureHealthRequired => "erasure_health_required",
Self::MissingCoverage => "missing_coverage",
Self::CachedFailures => "cached_failures",
Self::UncompactedChild => "uncompacted_child",
Self::SelectedForWalk => "selected_for_walk",
Self::Eligible => "eligible",
}
}
fn not_evaluated(self) -> bool {
matches!(
self,
Self::MissingScope | Self::NoResumeSweep | Self::DeepScan | Self::ForwardSweep | Self::ActiveReplication
)
}
}
/// Scope admission and end-of-cycle evidence are separate observations. An
/// activated completion proof does not mean this cycle used prefix reuse.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerPrefixReuseStage {
ScopeResolution,
CycleCompletion,
}
/// The first existing baseline validation check that rejected a snapshot.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum ScannerBaselineRejection {
Missing,
Decode,
Incomplete,
Partial,
Marks,
Epoch,
SourceCount,
PlanMissing,
PlanMismatch,
SourceIndex,
SourceMembership,
SourceDuplicate,
SetIncomplete,
SetTombstone,
SetEpoch,
SetCycle,
SetPlan,
SourceCoverage,
AuthoritativeIdentity,
NotNewer,
}
impl ScannerBaselineRejection {
const ALL: [Self; 20] = [
Self::Missing,
Self::Decode,
Self::Incomplete,
Self::Partial,
Self::Marks,
Self::Epoch,
Self::SourceCount,
Self::PlanMissing,
Self::PlanMismatch,
Self::SourceIndex,
Self::SourceMembership,
Self::SourceDuplicate,
Self::SetIncomplete,
Self::SetTombstone,
Self::SetEpoch,
Self::SetCycle,
Self::SetPlan,
Self::SourceCoverage,
Self::AuthoritativeIdentity,
Self::NotNewer,
];
fn label(self) -> &'static str {
match self {
Self::Missing => "missing",
Self::Decode => "decode",
Self::Incomplete => "incomplete",
Self::Partial => "partial",
Self::Marks => "marks",
Self::Epoch => "epoch",
Self::SourceCount => "source_count",
Self::PlanMissing => "plan_missing",
Self::PlanMismatch => "plan_mismatch",
Self::SourceIndex => "source_index",
Self::SourceMembership => "source_membership",
Self::SourceDuplicate => "source_duplicate",
Self::SetIncomplete => "set_incomplete",
Self::SetTombstone => "set_tombstone",
Self::SetEpoch => "set_epoch",
Self::SetCycle => "set_cycle",
Self::SetPlan => "set_plan",
Self::SourceCoverage => "source_coverage",
Self::AuthoritativeIdentity => "authoritative_identity",
Self::NotNewer => "not_newer",
}
}
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerMetadataReadSnapshot {
pub attempts: u64,
pub succeeded: u64,
pub not_found: u64,
pub errors: u64,
pub cancelled: u64,
/// Bytes returned by successful storage reads, including corrupt metadata.
pub bytes: u64,
/// Elapsed read-future time, including failed and cancelled reads.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseSnapshot {
/// Completed subtree reuse decisions; not cache entries or objects skipped.
pub count: u64,
/// Time spent copying and attaching the reused subtree, excluding listing.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerBucketReuseSnapshot {
/// Cold bucket copies or current roots successfully handed to the collector.
pub count: u64,
/// Cold copy/attachment time or current-root channel-send time; excludes validation.
pub elapsed_nanos: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerCacheReuseDecisionSnapshot {
/// Policy prerequisites that stopped the gate before cache proof checks.
pub not_evaluated: BTreeMap<String, u64>,
/// First failed cache/work condition; later conditions are not evaluated.
pub blocked: BTreeMap<String, u64>,
/// Admitted candidates, which may still need EC classification before copying.
pub eligible: u64,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerPrefixReuseSnapshot {
pub evaluated: u64,
pub activated: u64,
/// All failed checks are counted; one evaluation can have several blockers.
pub blockers: BTreeMap<String, u64>,
}
/// Missing on an older node means unobserved, not zero work or healthy reuse.
#[derive(Clone, Debug, Default, Serialize, Deserialize)]
#[serde(default)]
pub struct ScannerScanObservationSnapshot {
pub metadata_reads: ScannerMetadataReadSnapshot,
pub cache_reuse: BTreeMap<String, ScannerCacheReuseSnapshot>,
pub bucket_reuse: BTreeMap<String, ScannerBucketReuseSnapshot>,
pub reuse_decisions: BTreeMap<String, ScannerCacheReuseDecisionSnapshot>,
/// Clean root children still walked because EC scans also check health.
pub erasure_clean_prefix_walks: u64,
pub prefix_scope_resolution: ScannerPrefixReuseSnapshot,
pub prefix_cycle_completion: ScannerPrefixReuseSnapshot,
pub scope_fallbacks: BTreeMap<String, u64>,
/// Rejections only for cycles with a baseline_unavailable scope fallback.
pub authoritative_baseline_rejections: BTreeMap<String, u64>,
pub observed_baseline_rejections: BTreeMap<String, u64>,
}
#[derive(Default)]
pub struct ScannerScanObservation {
metadata: [AtomicU64; 7],
cache_count: [AtomicU64; 4],
cache_nanos: [AtomicU64; 4],
bucket_count: [AtomicU64; 2],
bucket_nanos: [AtomicU64; 2],
reuse_decisions: [[AtomicU64; ScannerCacheReuseDecision::ALL.len()]; 3],
erasure_clean_prefix_walks: AtomicU64,
prefix_evaluated: [AtomicU64; 2],
prefix_activated: [AtomicU64; 2],
prefix_blockers: [[AtomicU64; PREFIX_REUSE_BLOCKERS.len()]; 2],
scope_fallbacks: [AtomicU64; SCOPE_FALLBACK_REASONS.len()],
baseline_rejections: [[AtomicU64; ScannerBaselineRejection::ALL.len()]; 2],
}
fn duration_nanos(duration: Duration) -> u64 {
u64::try_from(duration.as_nanos()).unwrap_or(u64::MAX)
}
fn labelled_counts(labels: impl IntoIterator<Item = &'static str>, counters: &[AtomicU64]) -> BTreeMap<String, u64> {
labels
.into_iter()
.zip(counters)
.map(|(label, count)| (label.to_owned(), count.load(Ordering::Relaxed)))
.collect()
}
impl ScannerScanObservation {
pub(crate) fn start_metadata_read(&self) {
self.metadata[0].fetch_add(1, Ordering::Relaxed);
}
pub(crate) fn finish_metadata_read(&self, outcome: ScannerMetadataReadOutcome, bytes: u64, elapsed: Duration) {
self.metadata[1 + outcome as usize].fetch_add(1, Ordering::Relaxed);
if outcome == ScannerMetadataReadOutcome::Success {
self.metadata[5].fetch_add(bytes, Ordering::Relaxed);
}
self.metadata[6].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
pub fn record_cache_reuse(&self, kind: ScannerCacheReuseKind, elapsed: Duration) {
self.cache_count[kind as usize].fetch_add(1, Ordering::Relaxed);
self.cache_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records completed reuse work only; it grants no publication authority.
pub fn record_bucket_reuse(&self, kind: ScannerBucketReuseKind, count: u64, elapsed: Duration) {
self.bucket_count[kind as usize].fetch_add(count, Ordering::Relaxed);
self.bucket_nanos[kind as usize].fetch_add(duration_nanos(elapsed), Ordering::Relaxed);
}
/// Records one first-condition decision without inspecting later prerequisites.
pub fn record_cache_reuse_decision(&self, gate: ScannerCacheReuseGate, decision: ScannerCacheReuseDecision) {
self.reuse_decisions[gate as usize][decision as usize].fetch_add(1, Ordering::Relaxed);
}
pub fn record_erasure_clean_prefix_walk(&self) {
self.erasure_clean_prefix_walks.fetch_add(1, Ordering::Relaxed);
}
pub fn record_prefix_reuse(&self, stage: ScannerPrefixReuseStage, activated: bool, blockers: &[Option<&str>; 8]) {
let stage = stage as usize;
self.prefix_evaluated[stage].fetch_add(1, Ordering::Relaxed);
if activated {
self.prefix_activated[stage].fetch_add(1, Ordering::Relaxed);
}
for blocker in blockers.iter().flatten() {
if let Some(index) = PREFIX_REUSE_BLOCKERS.iter().position(|label| label == blocker) {
self.prefix_blockers[stage][index].fetch_add(1, Ordering::Relaxed);
}
}
}
pub(crate) fn record_scope_fallback(&self, reason: &str) {
if let Some(index) = SCOPE_FALLBACK_REASONS.iter().position(|label| *label == reason) {
self.scope_fallbacks[index].fetch_add(1, Ordering::Relaxed);
}
}
pub fn record_baseline_rejection(&self, authoritative: ScannerBaselineRejection, observed: Option<ScannerBaselineRejection>) {
self.baseline_rejections[0][authoritative as usize].fetch_add(1, Ordering::Relaxed);
if let Some(observed) = observed {
self.baseline_rejections[1][observed as usize].fetch_add(1, Ordering::Relaxed);
}
}
pub fn snapshot(&self) -> ScannerScanObservationSnapshot {
let prefix = |stage: ScannerPrefixReuseStage| ScannerPrefixReuseSnapshot {
evaluated: self.prefix_evaluated[stage as usize].load(Ordering::Relaxed),
activated: self.prefix_activated[stage as usize].load(Ordering::Relaxed),
blockers: labelled_counts(PREFIX_REUSE_BLOCKERS, &self.prefix_blockers[stage as usize]),
};
ScannerScanObservationSnapshot {
metadata_reads: ScannerMetadataReadSnapshot {
attempts: self.metadata[0].load(Ordering::Relaxed),
succeeded: self.metadata[1].load(Ordering::Relaxed),
not_found: self.metadata[2].load(Ordering::Relaxed),
errors: self.metadata[3].load(Ordering::Relaxed),
cancelled: self.metadata[4].load(Ordering::Relaxed),
bytes: self.metadata[5].load(Ordering::Relaxed),
elapsed_nanos: self.metadata[6].load(Ordering::Relaxed),
},
cache_reuse: ScannerCacheReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerCacheReuseSnapshot {
count: self.cache_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.cache_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
bucket_reuse: ScannerBucketReuseKind::ALL
.into_iter()
.map(|kind| {
(
kind.label().to_owned(),
ScannerBucketReuseSnapshot {
count: self.bucket_count[kind as usize].load(Ordering::Relaxed),
elapsed_nanos: self.bucket_nanos[kind as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
reuse_decisions: ScannerCacheReuseGate::ALL
.into_iter()
.map(|gate| {
let counts = &self.reuse_decisions[gate as usize];
let decisions = |not_evaluated| {
ScannerCacheReuseDecision::ALL
.into_iter()
.filter(|decision| {
*decision != ScannerCacheReuseDecision::Eligible && decision.not_evaluated() == not_evaluated
})
.map(|decision| (decision.label().to_owned(), counts[decision as usize].load(Ordering::Relaxed)))
.collect()
};
(
gate.label().to_owned(),
ScannerCacheReuseDecisionSnapshot {
not_evaluated: decisions(true),
blocked: decisions(false),
eligible: counts[ScannerCacheReuseDecision::Eligible as usize].load(Ordering::Relaxed),
},
)
})
.collect(),
erasure_clean_prefix_walks: self.erasure_clean_prefix_walks.load(Ordering::Relaxed),
prefix_scope_resolution: prefix(ScannerPrefixReuseStage::ScopeResolution),
prefix_cycle_completion: prefix(ScannerPrefixReuseStage::CycleCompletion),
scope_fallbacks: labelled_counts(SCOPE_FALLBACK_REASONS, &self.scope_fallbacks),
authoritative_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[0],
),
observed_baseline_rejections: labelled_counts(
ScannerBaselineRejection::ALL.map(ScannerBaselineRejection::label),
&self.baseline_rejections[1],
),
}
}
}
/// Dropping an unfinished read records cancellation. The guard observes the
/// read future only; it cannot cancel it or change its storage result.
#[must_use]
pub struct ScannerMetadataReadGuard<'a> {
pub(crate) metrics: &'a Metrics,
pub(crate) started: Instant,
pub(crate) outcome: ScannerMetadataReadOutcome,
pub(crate) bytes: u64,
}
impl ScannerMetadataReadGuard<'_> {
pub fn finish(mut self, outcome: ScannerMetadataReadOutcome, bytes: u64) {
self.outcome = outcome;
self.bytes = bytes;
}
}
impl Drop for ScannerMetadataReadGuard<'_> {
fn drop(&mut self) {
self.metrics
.finish_scanner_metadata_read(self.outcome, self.bytes, self.started.elapsed());
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::Arc;
#[tokio::test]
async fn metadata_read_reports_storage_results_and_existing_read_metric() {
let metrics = Metrics::new();
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 127);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Success, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::NotFound, 0);
metrics
.start_scanner_metadata_read()
.finish(ScannerMetadataReadOutcome::Error, 0);
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!(
(reads.attempts, reads.succeeded, reads.not_found, reads.errors, reads.cancelled),
(4, 2, 1, 1, 0)
);
assert_eq!(reads.bytes, 127);
assert_eq!(report.life_time_ops["read_metadata"], 4);
assert_eq!(report.last_minute.actions["read_metadata"].count, 4);
assert_eq!(report.last_minute.actions["read_metadata"].bytes, 127);
}
#[tokio::test]
async fn metadata_read_cancellation_balances_an_actually_polled_future_once() {
let metrics = Arc::new(Metrics::new());
let worker_metrics = Arc::clone(&metrics);
let (started, admitted) = tokio::sync::oneshot::channel();
let worker = tokio::spawn(async move {
let read = worker_metrics.start_scanner_metadata_read();
started.send(()).expect("observe started read");
std::future::pending::<()>().await;
read.finish(ScannerMetadataReadOutcome::Success, 99);
});
tokio::time::timeout(Duration::from_secs(5), admitted)
.await
.expect("read starts within deadline")
.expect("read started");
let in_flight = metrics.scan_observation().snapshot().metadata_reads;
assert_eq!((in_flight.attempts, in_flight.cancelled, in_flight.succeeded), (1, 0, 0));
worker.abort();
assert!(worker.await.expect_err("read task was aborted").is_cancelled());
let report = metrics.report().await;
let reads = &report.scan_observation.metadata_reads;
assert_eq!((reads.attempts, reads.cancelled, reads.succeeded, reads.bytes), (1, 1, 0, 0));
assert_eq!(report.life_time_ops["read_metadata"], 1);
assert_eq!(report.last_minute.actions["read_metadata"].count, 1);
assert_eq!(
Duration::from_nanos(reads.elapsed_nanos).as_secs(),
report.last_minute.actions["read_metadata"].acc_time
);
}
#[tokio::test]
async fn observation_report_keeps_stages_reasons_and_copy_costs_separate() {
let metrics = Metrics::new();
let observation = metrics.scan_observation();
observation.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, Duration::from_nanos(321));
observation.record_bucket_reuse(ScannerBucketReuseKind::ScopedColdBucket, 2, Duration::from_nanos(654));
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, Duration::from_nanos(987));
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::MissingScope);
observation
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, ScannerCacheReuseDecision::ErasureHealthRequired);
observation.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, ScannerCacheReuseDecision::Eligible);
observation.record_erasure_clean_prefix_walk();
observation.record_prefix_reuse(
ScannerPrefixReuseStage::ScopeResolution,
false,
&[
Some("missing_producer_identity"),
Some("missing_durable_journal_replay"),
None,
None,
None,
None,
None,
None,
],
);
observation.record_prefix_reuse(ScannerPrefixReuseStage::CycleCompletion, true, &[None; 8]);
metrics.record_scanner_scope_fallback("baseline_unavailable");
observation.record_baseline_rejection(ScannerBaselineRejection::Epoch, Some(ScannerBaselineRejection::Missing));
let report = metrics.report().await;
let snapshot = &report.scan_observation;
assert_eq!(snapshot.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.cache_reuse["clean_prefix"].elapsed_nanos, 321);
assert_eq!(snapshot.cache_reuse["sampled_namespace"].count, 0);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].count, 2);
assert_eq!(snapshot.bucket_reuse["scoped_cold_bucket"].elapsed_nanos, 654);
assert_eq!(snapshot.bucket_reuse["current_root"].count, 1);
assert_eq!(snapshot.bucket_reuse["current_root"].elapsed_nanos, 987);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"], 1);
assert_eq!(snapshot.reuse_decisions["clean_prefix"].blocked["erasure_health_required"], 1);
assert_eq!(snapshot.reuse_decisions["covered_namespace"].eligible, 1);
let mut prior_observations = serde_json::to_value(snapshot).expect("encode observations");
let fields = prior_observations.as_object_mut().expect("observation object");
fields.remove("bucket_reuse");
fields.remove("reuse_decisions");
let prior: ScannerScanObservationSnapshot =
serde_json::from_value(prior_observations).expect("read earlier observations");
assert!(prior.bucket_reuse.is_empty());
assert!(prior.reuse_decisions.is_empty());
assert_eq!(prior.cache_reuse["clean_prefix"].count, 1);
assert_eq!(snapshot.erasure_clean_prefix_walks, 1);
assert_eq!(
(snapshot.prefix_scope_resolution.evaluated, snapshot.prefix_scope_resolution.activated),
(1, 0)
);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_producer_identity"], 1);
assert_eq!(snapshot.prefix_scope_resolution.blockers["missing_durable_journal_replay"], 1);
assert_eq!(
(snapshot.prefix_cycle_completion.evaluated, snapshot.prefix_cycle_completion.activated),
(1, 1)
);
assert_eq!(
snapshot.scope_fallbacks["baseline_unavailable"],
report.usage_freshness.scope_fallback_total
);
assert_eq!(snapshot.authoritative_baseline_rejections["epoch"], 1);
assert_eq!(snapshot.observed_baseline_rejections["missing"], 1);
let mut legacy = serde_json::to_value(&report).expect("encode report");
legacy
.as_object_mut()
.expect("report is an object")
.remove("scan_observation");
let decoded: crate::metrics::ScannerMetricsReport = serde_json::from_value(legacy).expect("read legacy report");
assert!(decoded.scan_observation.cache_reuse.is_empty(), "legacy counters are unobserved");
assert_eq!(decoded.scan_observation.metadata_reads.attempts, 0);
}
}
-26
View File
@@ -206,32 +206,6 @@ pub(crate) struct DataUsageCacheRevisions {
backup: Option<DataUsageCacheRevision>,
}
/// In-memory provenance of the decoded cache and the revisions observed by its loader.
#[derive(Clone, Debug)]
pub(crate) struct DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions,
loaded_from_main: bool,
}
impl DataUsageCacheBaselineRevisions {
pub(crate) fn revisions(&self) -> &DataUsageCacheRevisions {
&self.revisions
}
/// A backup may catch up to an unchanged primary without changing the decoded baseline.
/// Both reads must have decoded that primary; an ETag on a corrupt primary is insufficient.
pub(crate) fn permits_unchanged_baseline_since(&self, initial: &Self) -> bool {
if self.revisions == initial.revisions {
return true;
}
self.loaded_from_main
&& initial.loaded_from_main
&& matches!(&initial.revisions.main, DataUsageCacheRevision::Etag(etag) if !etag.is_empty())
&& self.revisions.main == initial.revisions.main
&& self.revisions.backup.as_ref() == Some(&self.revisions.main)
}
}
pub static DATA_USAGE_BUCKET: LazyLock<String> =
LazyLock::new(|| format!("{RUSTFS_META_BUCKET}{SLASH_SEPARATOR}{BUCKET_META_PREFIX}"));
@@ -70,7 +70,6 @@ struct DataUsageCacheLoadResult {
cache: DataUsageCache,
main_revision: DataUsageCacheRevision,
backup_revision: Option<DataUsageCacheRevision>,
loaded_from_main: bool,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -107,18 +106,6 @@ impl DataUsageCache {
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheRevisions> {
self.load_with_baseline_revisions(store, name)
.await
.map(|baseline| baseline.revisions)
}
/// Loads cache revisions with provenance for unchanged-primary backup reconciliation.
/// Missing or rebuilt caches and a decoded backup do not establish a decoded primary.
pub(crate) async fn load_with_baseline_revisions<S: ScannerObjectIO>(
&mut self,
store: Arc<S>,
name: &str,
) -> StorageResult<DataUsageCacheBaselineRevisions> {
let backup_name = format!("{name}.bkp");
let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]);
let loaded = Self::load_cache(store.clone(), name).await?;
@@ -146,10 +133,7 @@ impl DataUsageCache {
let main = loaded.main_revision;
*self = loaded.cache;
Ok(DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions { main, backup },
loaded_from_main: loaded.loaded_from_main,
})
Ok(DataUsageCacheRevisions { main, backup })
}
pub(crate) async fn read_revisions<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheRevisions> {
@@ -172,13 +156,10 @@ impl DataUsageCache {
cache,
revision: Some(main_revision),
} => {
// Legacy-location fallback carries Missing for the primary path.
let loaded_from_main = matches!(&main_revision, DataUsageCacheRevision::Etag(_));
return Ok(DataUsageCacheLoadResult {
cache: *cache,
main_revision,
backup_revision: None,
loaded_from_main,
});
}
DataUsageCacheLoadAttempt::Loaded { revision: None, .. } => {
@@ -216,7 +197,6 @@ impl DataUsageCache {
cache: *cache,
main_revision,
backup_revision: Some(backup_revision),
loaded_from_main: false,
});
}
DataUsageCacheLoadState::Loaded
@@ -240,7 +220,6 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache missing state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
(DataUsageCacheLoadState::Corrupt, DataUsageCacheLoadState::Missing)
@@ -260,7 +239,6 @@ impl DataUsageCache {
main_revision: main_revision
.ok_or_else(|| Error::other(format!("scanner cache corrupt state has no revision: {name}")))?,
backup_revision,
loaded_from_main: false,
});
}
_ => {}
@@ -772,102 +772,6 @@ async fn corrupt_primary_cache_loads_valid_backup() {
assert_eq!(store.puts.load(Ordering::SeqCst), 0);
}
#[test]
fn baseline_revisions_allow_only_a_decoded_unchanged_primary_backup_catchup() {
let etag = |value: &str| DataUsageCacheRevision::Etag(value.to_owned());
let initial = DataUsageCacheBaselineRevisions {
revisions: DataUsageCacheRevisions {
main: etag("primary"),
backup: Some(etag("old-backup")),
},
loaded_from_main: true,
};
assert!(initial.permits_unchanged_baseline_since(&initial));
let mut caught_up = initial.clone();
caught_up.revisions.backup = Some(etag("primary"));
assert!(caught_up.permits_unchanged_baseline_since(&initial));
assert!(
!initial.permits_unchanged_baseline_since(&caught_up),
"backup rollback changes the baseline"
);
for backup in [None, Some(DataUsageCacheRevision::Missing)] {
let mut missing_backup = initial.clone();
missing_backup.revisions.backup = backup;
assert!(caught_up.permits_unchanged_baseline_since(&missing_backup));
assert!(!missing_backup.permits_unchanged_baseline_since(&initial));
}
for (initial_from_main, current_from_main) in [(false, true), (true, false), (false, false)] {
let mut baseline = initial.clone();
baseline.loaded_from_main = initial_from_main;
let mut current = caught_up.clone();
current.loaded_from_main = current_from_main;
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"backup fallback grants no primary proof"
);
current.revisions = baseline.revisions.clone();
assert!(
current.permits_unchanged_baseline_since(&baseline),
"exact revisions retain existing admission"
);
}
let mut changed_main = caught_up.clone();
changed_main.revisions.main = etag("new-primary");
changed_main.revisions.backup = Some(etag("new-primary"));
assert!(!changed_main.permits_unchanged_baseline_since(&initial));
let mut changed_backup = caught_up;
changed_backup.revisions.backup = Some(etag("unrelated-backup"));
assert!(!changed_backup.permits_unchanged_baseline_since(&initial));
for main in [DataUsageCacheRevision::Missing, etag("")] {
let mut baseline = initial.clone();
baseline.revisions.main = main.clone();
let mut current = baseline.clone();
current.revisions.backup = Some(main);
assert!(
!current.permits_unchanged_baseline_since(&baseline),
"missing or empty primary ETag is not proof"
);
}
}
#[tokio::test]
async fn baseline_revisions_record_the_actual_decoded_cache_source() {
let mut expected = DataUsageCache::default();
expected.info.name = "decoded".to_owned();
let valid = expected.marshal_msg().expect("encode valid cache");
for (main, backup, from_main, name) in [
(valid.clone(), Some(valid.clone()), true, "decoded"),
(b"corrupt".to_vec(), Some(valid.clone()), false, "decoded"),
(b"corrupt".to_vec(), None, false, ""),
(Vec::new(), None, false, ""),
] {
let store = Arc::new(CacheReadStore::new(CacheReadBody::Bytes(main), backup));
let mut loaded = DataUsageCache::default();
let baseline = loaded
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
.expect("load actual cache source");
assert_eq!(baseline.loaded_from_main, from_main);
assert_eq!(loaded.info.name, name);
let mut raw_loaded = DataUsageCache::default();
let raw = raw_loaded
.load_with_revisions(store, DATA_USAGE_CACHE_NAME)
.await
.expect("load compatible raw revisions");
assert_eq!(baseline.revisions(), &raw, "raw loader retains its revision contract");
}
let mut legacy_store = CacheReadStore::new(CacheReadBody::Bytes(Vec::new()), None);
legacy_store.legacy = Some(valid);
let mut legacy_cache = DataUsageCache::default();
let legacy = legacy_cache
.load_with_baseline_revisions(Arc::new(legacy_store), "legacy-cache")
.await
.expect("load cache from the legacy location");
assert_eq!(legacy_cache.info.name, "decoded");
assert!(!legacy.loaded_from_main);
assert_eq!(legacy.revisions.main, DataUsageCacheRevision::Missing);
}
#[tokio::test]
async fn corrupt_cache_without_valid_backup_rebuilds_with_cas_revisions() {
for backup in [None, Some(b"also-not-msgpack".to_vec())] {
+9 -8
View File
@@ -248,13 +248,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2;
/// otherwise idle instance whose clean-idle backoff had doubled a 60 s
/// interval), which defeats the fast path it is meant to protect.
///
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] protects
/// against a persistently hot bucket driving an unbroken full-scan loop, so
/// retries can start small: 5 s, 10 s, 20 s … Superseded retries cap this
/// schedule at the ordinary cycle interval and preserve an operator-selected
/// nominal cadence. Preflight deferrals retain the generic
/// [`SCANNER_RETRY_MAX_INTERVAL`] cap so a temporarily unavailable peer cannot
/// drive a tight retry loop.
/// The exponential growth in [`ScannerRetryBackoff::retry_interval`] is
/// what protects against a persistently hot bucket driving an unbroken
/// full-scan loop, so it can start small: 5 s, 10 s, 20 s … capped by
/// [`SCANNER_RETRY_MAX_INTERVAL`]. A one-off race recovers in seconds; a
/// genuinely hot bucket still reaches minute-scale backoff within a handful of
/// cycles. Preflight deferrals use the same bounded schedule so a temporarily
/// unavailable peer cannot drive a tight retry loop.
const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5);
const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60);
/// Probe cadence while data movement pauses the scanner. Recovery retries use
@@ -3528,7 +3528,8 @@ where
));
let mut wait_plan =
scanner_cycle_wait_plan(&runtime_config, clean_idle_backoff, backoff_enabled, randomized_cycle_delay_for);
let superseded_retry_interval = scanner_superseded_retry_interval(superseded_backoff, &runtime_config);
let superseded_retry_interval =
scanner_superseded_retry_interval(superseded_backoff, &runtime_config, maintenance_features);
let deferred_retry_interval = deferred_backoff.retry_interval(runtime_config.cycle_interval);
let scope_fallback_retry_interval = scanner_scope_fallback_retry_interval(scope_fallback_backoff, &runtime_config);
let mut convergence_retry_interval = superseded_retry_interval
+10 -7
View File
@@ -192,16 +192,19 @@ impl ScannerRetryBackoff {
pub(super) fn scanner_superseded_retry_interval(
backoff: ScannerRetryBackoff,
runtime_config: &ScannerRuntimeConfig,
maintenance_features: ScannerMaintenanceFeatures,
) -> Option<Duration> {
let cycle_interval = runtime_config.cycle_interval.max(Duration::from_secs(1));
let retry_interval = backoff.retry_interval(cycle_interval)?;
// Preserve an operator-selected nominal cadence during convergence.
let mut retry_interval = backoff.retry_interval(runtime_config.cycle_interval)?;
// Preserve an operator-selected duty-cycle floor during convergence.
if runtime_config.cycle_interval_source != ScannerRuntimeConfigSource::Default {
return Some(cycle_interval);
retry_interval = retry_interval.max(runtime_config.cycle_interval);
}
// A complete but non-authoritative walk must not postpone convergence
// beyond the ordinary cadence, even without lifecycle or replication.
Some(retry_interval.min(cycle_interval))
// A complete but non-authoritative walk must not postpone lifecycle or
// replication evaluation beyond its ordinary maintenance cadence.
if maintenance_features.needs_regular_cycle() {
retry_interval = retry_interval.min(runtime_config.cycle_interval.max(Duration::from_secs(1)));
}
Some(retry_interval)
}
/// Pacing for the cycle after a full-namespace scope fallback while dirty
+38 -73
View File
@@ -8947,7 +8947,7 @@ fn clean_idle_backoff_grows_to_cap() {
}
#[test]
fn retry_backoff_grows_caps_and_resets_after_convergence() {
fn superseded_retry_backoff_grows_caps_and_resets_after_convergence() {
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(backoff.retry_interval(Duration::from_secs(24 * 60 * 60)), None);
@@ -8971,7 +8971,7 @@ fn retry_backoff_grows_caps_and_resets_after_convergence() {
}
#[test]
fn retry_backoff_respects_a_faster_configured_cycle() {
fn superseded_retry_backoff_respects_a_faster_configured_cycle() {
let mut backoff = ScannerRetryBackoff::default();
backoff.record_retryable_cycle(true);
@@ -8983,7 +8983,7 @@ fn retry_backoff_respects_a_faster_configured_cycle() {
}
#[test]
fn retry_backoff_grows_from_the_default_cycle() {
fn superseded_retry_backoff_grows_from_the_default_cycle() {
let mut backoff = ScannerRetryBackoff::default();
// The first race after a write burst retries in seconds, not a whole
// cycle, while repeated supersedes still climb toward the cap.
@@ -9003,7 +9003,7 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &default_config),
scanner_superseded_retry_interval(backoff, &default_config, ScannerMaintenanceFeatures::default()),
Some(SCANNER_RETRY_BASE_INTERVAL),
"the default adaptive cadence must retain fast convergence"
);
@@ -9013,24 +9013,24 @@ fn superseded_retry_preserves_explicit_cycle_cadence() {
ScannerRuntimeConfigSource::Config,
ScannerRuntimeConfigSource::ScannerCompatConfig,
] {
for cycle_secs in [1, 3, 600, 6 * 60 * 60] {
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
cycle_interval_source: source,
let runtime_config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(6 * 60 * 60),
cycle_interval_source: source,
..Default::default()
};
for features in [
ScannerMaintenanceFeatures::default(),
ScannerMaintenanceFeatures {
lifecycle: true,
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
for _ in 0..40 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must hold throughout convergence retries"
);
}
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &runtime_config), None);
},
] {
assert_eq!(
scanner_superseded_retry_interval(backoff, &runtime_config, features),
Some(runtime_config.cycle_interval),
"{source:?} cycle cadence must not be shortened by convergence retries"
);
}
}
}
@@ -9051,7 +9051,7 @@ fn superseded_retry_does_not_delay_required_maintenance() {
for consecutive_cycles in 1..=12 {
backoff.record_retryable_cycle(true);
assert_eq!(
scanner_superseded_retry_interval(backoff, &config),
scanner_superseded_retry_interval(backoff, &config, features),
Some(config.cycle_interval),
"superseded cycle {consecutive_cycles} must preserve regular maintenance"
);
@@ -9137,50 +9137,19 @@ fn scope_fallback_retry_preserves_explicit_and_fast_cycle_cadence() {
}
#[test]
fn superseded_retry_caps_at_the_default_cycle_and_resets_after_convergence() {
fn superseded_retry_only_caps_regular_maintenance() {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
for expected in [5, 10, 20, 40, 60, 60, 60, 60] {
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(Duration::from_secs(expected)));
}
backoff.consecutive_cycles = u32::MAX;
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(true);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), Some(config.cycle_interval));
backoff.record_retryable_cycle(false);
assert_eq!(scanner_superseded_retry_interval(backoff, &config), None);
}
#[test]
fn superseded_retry_caps_fast_cycles_without_changing_the_generic_retry_cap() {
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..40 {
for _ in 0..6 {
backoff.record_retryable_cycle(true);
}
assert_eq!(
backoff.retry_interval(Duration::from_secs(60)),
Some(Duration::from_secs(30 * 60)),
"deferred and transient retries must retain the generic cap"
scanner_superseded_retry_interval(backoff, &config, ScannerMaintenanceFeatures::default()),
Some(Duration::from_secs(160)),
);
for cycle_secs in [0, 1, 3, 60, 6 * 60 * 60] {
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(cycle_secs),
..Default::default()
};
assert_eq!(
scanner_superseded_retry_interval(backoff, &config),
Some(Duration::from_secs(cycle_secs.max(1))),
);
}
}
#[test]
fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
for features in [
ScannerMaintenanceFeatures {
lifecycle: true,
@@ -9195,23 +9164,28 @@ fn superseded_retry_timer_keeps_required_maintenance_full_scans() {
..Default::default()
},
] {
assert!(features.requires_full_scan(Some(7), 7, ScannerCycleWakeReason::Timer, true));
assert_eq!(scanner_superseded_retry_interval(backoff, &config, features), Some(config.cycle_interval));
assert_eq!(scanner_superseded_retry_interval(ScannerRetryBackoff::default(), &config, features), None);
}
}
#[tokio::test(start_paused = true)]
#[serial]
async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle() {
async fn superseded_maintenance_wait_keeps_convergence_fencing() {
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
let config = ScannerRuntimeConfig {
cycle_interval: Duration::from_secs(60),
..Default::default()
};
let features = ScannerMaintenanceFeatures {
lifecycle: true,
..Default::default()
};
let mut backoff = ScannerRetryBackoff::default();
for _ in 0..12 {
backoff.record_retryable_cycle(true);
}
let interval = scanner_superseded_retry_interval(backoff, &config);
let interval = scanner_superseded_retry_interval(backoff, &config, features);
assert_eq!(interval, Some(config.cycle_interval));
let generations = ScannerCycleObservedGenerations::for_wait(
&config,
@@ -9226,28 +9200,19 @@ async fn superseded_retry_wait_keeps_convergence_fencing_at_the_default_cycle()
let mut seen = Some(BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 7, 3))]));
let changed = BTreeMap::from([("node-2".to_string(), scanner_node_activity("epoch-a", 8, 3))]);
let started = tokio::time::Instant::now();
let mut wait = Box::pin(wait_for_next_scanner_cycle_with_activity(
let reason = wait_for_next_scanner_cycle_with_activity(
&ctx,
interval.expect("superseded retry must retain its convergence interval"),
interval.expect("maintenance must retain its convergence interval"),
Some(Duration::from_secs(1)),
&mut seen,
generations,
|| false,
|| std::future::ready(Ok(changed.clone())),
));
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
for _ in 0..3 {
tokio::time::advance(Duration::from_secs(15)).await;
crate::scanner_io::record_dirty_usage_bucket("photos");
assert!(matches!(futures::poll!(&mut wait), Poll::Pending));
}
let reason = wait.await;
)
.await;
assert_eq!(reason, ScannerCycleWakeReason::Timer);
assert_eq!(started.elapsed(), config.cycle_interval);
assert_eq!(seen, Some(changed));
crate::scanner_io::clear_dirty_usage_buckets_for_tests();
}
#[test]
+44 -105
View File
@@ -59,7 +59,6 @@ use rustfs_scanner_metrics::metrics::{
CloseDiskGuard, IlmAction, Metric, Metrics, ScannerReplicationRepairKind, ScannerSourceWorkUpdate, ScannerWorkSource,
UpdateCurrentPathFn, current_path_updater, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{ScannerCacheReuseDecision, ScannerCacheReuseGate, ScannerCacheReuseKind};
use rustfs_utils::path::{SLASH_SEPARATOR, path_join_buf};
use sha2::{Digest, Sha256};
use time::OffsetDateTime;
@@ -1061,51 +1060,38 @@ impl FolderScanner {
child_hash: &DataUsageHash,
abandoned_children: &DataUsageHashMap,
) -> bool {
let decision = if let Some(prefix_scan_scope) = &self.prefix_scan_scope {
if folder.parent.is_some() || folder.name != self.old_cache.info.name {
ScannerCacheReuseDecision::NotRoot
} else if into.compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !abandoned_children.contains(&child_hash.0) {
ScannerCacheReuseDecision::MissingCachedChild
} else if let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
{
if entry.is_empty() || entry.contains('/') {
ScannerCacheReuseDecision::InvalidPrefix
} else if prefix_scan_scope.contains(entry) {
ScannerCacheReuseDecision::DirtyPrefix
} else if self.is_erasure_mode {
// EC scans must still perform independent object-health work.
global_metrics().scan_observation().record_erasure_clean_prefix_walk();
ScannerCacheReuseDecision::ErasureHealthRequired
} else {
ScannerCacheReuseDecision::Eligible
}
} else {
ScannerCacheReuseDecision::InvalidPrefix
}
} else {
ScannerCacheReuseDecision::MissingScope
let Some(prefix_scan_scope) = &self.prefix_scan_scope else {
return false;
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CleanPrefix, decision);
decision == ScannerCacheReuseDecision::Eligible
// Erasure-mode usage scans also perform probabilistic object-health
// work. Reusing a clean subtree here would silently suppress that
// independent maintenance path, so prefix reuse is limited to the
// non-erasure data-usage scanner.
if self.is_erasure_mode
|| folder.parent.is_some()
|| folder.name != self.old_cache.info.name
|| into.compacted
|| !abandoned_children.contains(&clone_cache_value(&child_hash.0))
{
return false;
}
let Some(entry) = child
.name
.strip_prefix(folder.name.as_str())
.and_then(|entry| entry.strip_prefix('/'))
else {
return false;
};
!entry.is_empty() && !entry.contains('/') && !prefix_scan_scope.contains(entry)
}
fn reuse_clean_root_child(&mut self, child: &CachedFolder, child_hash: &DataUsageHash, into: &mut DataUsageEntry) {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
if let Some(updates) = &mut self.usage_updates {
updates.cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
}
into.add_child(child_hash);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CleanPrefix, started.elapsed());
}
fn failed_retry_suppressed(&self, path: &str) -> bool {
@@ -1292,60 +1278,14 @@ impl FolderScanner {
}
fn can_reuse_covered_namespace(&self, name: &str, compacted: bool) -> bool {
let decision = if self.old_cache.info.scan_progress.is_none() {
ScannerCacheReuseDecision::NoResumeSweep
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
{
ScannerCacheReuseDecision::MissingCoverage
} else if self.old_cache.find(name).is_none() {
ScannerCacheReuseDecision::MissingCachedChild
} else if self.resume_subtree_has_failures(name) {
ScannerCacheReuseDecision::CachedFailures
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::CoveredNamespace, decision);
decision == ScannerCacheReuseDecision::Eligible
}
// None preserves work that cannot use sampling; false requests this cycle's rescan.
fn sampled_namespace_reuse(&self, hash: &DataUsageHash, compacted: bool, forward_sweep: bool) -> Option<bool> {
let decision = if self.scan_mode == HealScanMode::Deep {
ScannerCacheReuseDecision::DeepScan
} else if forward_sweep {
ScannerCacheReuseDecision::ForwardSweep
} else if self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
{
ScannerCacheReuseDecision::ActiveReplication
} else if compacted {
ScannerCacheReuseDecision::CompactedParent
} else if !self.old_cache.is_compacted(hash) {
ScannerCacheReuseDecision::UncompactedChild
} else if hash.mod_(self.old_cache.info.next_cycle as u32, data_usage_update_dir_cycles()) {
ScannerCacheReuseDecision::SelectedForWalk
} else {
ScannerCacheReuseDecision::Eligible
};
global_metrics()
.scan_observation()
.record_cache_reuse_decision(ScannerCacheReuseGate::NamespaceSampling, decision);
match decision {
ScannerCacheReuseDecision::Eligible => Some(true),
ScannerCacheReuseDecision::SelectedForWalk => Some(false),
_ => None,
}
self.old_cache.info.scan_progress.is_some()
&& !compacted
&& self
.resume_frontier
.as_deref()
.is_some_and(|frontier| scan_frontier_covers_path(name, frontier))
&& self.old_cache.find(name).is_some()
&& !self.resume_subtree_has_failures(name)
}
fn resume_subtree_has_failures(&self, folder: &str) -> bool {
@@ -2137,11 +2077,7 @@ impl FolderScanner {
// The caller stitches this cached child into its
// parent; do not clone a growing child set into the
// unused provisional accumulator as well.
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &this_hash, &folder.parent);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::ClassifiedNamespace, started.elapsed());
self.budget.record_entries_visited(pending_entry_progress);
self.finish_raw_enumeration_parent(&folder.name);
done_folder();
@@ -2728,12 +2664,8 @@ impl FolderScanner {
FolderScanMode::Namespace
};
if child_mode == FolderScanMode::Namespace && reuse_covered {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::CoveredNamespace, started.elapsed());
continue;
}
@@ -2773,15 +2705,22 @@ impl FolderScanner {
}
FolderScanSource::Existing => {
// Usage sampling is not proof that a Deep check ran.
if let Some(reuse) = self.sampled_namespace_reuse(&h, into.compacted, forward_sweep) {
if reuse {
if self.scan_mode != HealScanMode::Deep
&& !forward_sweep
&& !self
.old_cache
.info
.replication
.as_ref()
.is_some_and(|config| config.has_active_rules("", true))
&& !into.compacted
&& self.old_cache.is_compacted(&h)
{
let next_cycle = self.old_cache.info.next_cycle as u32;
if !h.mod_(next_cycle, data_usage_update_dir_cycles()) {
if child_mode == FolderScanMode::Namespace {
let started = Instant::now();
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
into.add_child(&h);
global_metrics()
.scan_observation()
.record_cache_reuse(ScannerCacheReuseKind::SampledNamespace, started.elapsed());
self.record_scan_resume_hint(&folder_item.name);
continue;
}
-191
View File
@@ -3103,112 +3103,6 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Directories));
}
#[tokio::test]
#[serial]
async fn cache_reuse_gates_report_only_the_first_production_condition() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
scanner.old_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
compacted: true,
objects: 2,
..Default::default()
},
);
let hash = hash_path("bucket/child");
let folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
let child = CachedFolder {
name: "bucket/child".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
};
let abandoned = HashSet::from([hash.0.clone()]);
let before = global_metrics().scan_observation().snapshot();
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let missing_scope = global_metrics().scan_observation().snapshot();
assert_eq!(
missing_scope.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"],
before.reuse_decisions["clean_prefix"].not_evaluated["missing_scope"] + 1
);
scanner.prefix_scan_scope = ScannerBucketPrefixScanScope::from_dirty_top_level_entries(HashSet::from(["dirty".to_string()]));
scanner.is_erasure_mode = true;
assert!(!scanner.should_reuse_clean_root_child(&folder, &DataUsageEntry::default(), &child, &hash, &abandoned));
let erasure = global_metrics().scan_observation().snapshot();
assert_eq!(
erasure.reuse_decisions["clean_prefix"].blocked["erasure_health_required"],
before.reuse_decisions["clean_prefix"].blocked["erasure_health_required"] + 1
);
assert_eq!(erasure.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
assert!(!scanner.can_reuse_covered_namespace("bucket/child", true));
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_frontier = Some("bucket/child".to_string());
scanner.resume_failed_paths = vec!["bucket/child/failed".to_string()];
assert!(!scanner.can_reuse_covered_namespace("bucket/child", false));
scanner.resume_failed_paths.clear();
assert!(scanner.can_reuse_covered_namespace("bucket/child", false));
let covered = global_metrics().scan_observation().snapshot();
let reasons = &covered.reuse_decisions["covered_namespace"];
let prior = &before.reuse_decisions["covered_namespace"];
assert_eq!(reasons.not_evaluated["no_resume_sweep"], prior.not_evaluated["no_resume_sweep"] + 1);
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["missing_coverage"], prior.blocked["missing_coverage"] + 1);
assert_eq!(reasons.blocked["cached_failures"], prior.blocked["cached_failures"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
with_var(ENV_DATA_USAGE_UPDATE_DIR_CYCLES, Some("16"), || {
scanner.scan_mode = HealScanMode::Deep;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
scanner.scan_mode = HealScanMode::Normal;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, true), None);
let replication: ReplicationConfig = serde_json::from_value(serde_json::json!({
"config": { "role": "", "rules": [{ "destination": { "bucket": "arn:test" }, "id": "status", "prefix": "", "priority": 1, "status": "Enabled" }] },
"remotes": crate::storage_api::EcstoreBucketTargets::default(),
})).expect("active replication fixture");
scanner.old_cache.info.replication = Some(Arc::new(replication));
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
scanner.old_cache.info.replication = None;
assert_eq!(scanner.sampled_namespace_reuse(&hash, true, false), None);
let rescan_cycle = (0..16)
.find(|cycle| hash.mod_(*cycle, 16))
.expect("hash is visited in one sampling cycle");
scanner.old_cache.info.next_cycle = u64::from(rescan_cycle);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(false));
scanner.old_cache.info.next_cycle = u64::from((rescan_cycle + 1) % 16);
assert_eq!(scanner.sampled_namespace_reuse(&hash, false, false), Some(true));
});
let after = global_metrics().scan_observation().snapshot();
let reasons = &after.reuse_decisions["namespace_sampling"];
let prior = &before.reuse_decisions["namespace_sampling"];
for label in ["deep_scan", "forward_sweep", "active_replication"] {
assert_eq!(
reasons.not_evaluated[label],
prior.not_evaluated[label] + 1,
"first sampling condition {label}"
);
}
assert_eq!(reasons.blocked["compacted_parent"], prior.blocked["compacted_parent"] + 1);
assert_eq!(reasons.blocked["selected_for_walk"], prior.blocked["selected_for_walk"] + 1);
assert_eq!(reasons.eligible, prior.eligible + 1);
for (kind, reuse) in &after.cache_reuse {
assert_eq!(reuse.count, before.cache_reuse[kind].count, "gate admission alone does not copy {kind}");
}
}
#[tokio::test]
#[serial]
async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entries() {
@@ -3250,21 +3144,12 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
.await
.expect("scoped root scan should finish successfully");
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count + 1);
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks);
assert_eq!(
observed.reuse_decisions["clean_prefix"].eligible,
before.reuse_decisions["clean_prefix"].eligible + 1
);
let clean = scanner
.new_cache
.size_recursive("bucket/clean")
@@ -3282,78 +3167,6 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
assert_eq!((bucket.size, bucket.objects), (17, 3));
}
#[tokio::test]
#[serial]
async fn scoped_cache_observations_distinguish_checkpoint_and_periodic_sampling() {
for covered in [true, false] {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
tokio::fs::create_dir_all(temp_dir.join("bucket/child"))
.await
.expect("create cached namespace");
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/child",
"bucket",
DataUsageEntry {
size: 17,
objects: 5,
compacted: true,
..Default::default()
},
);
let kind = if covered {
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
scanner.resume_frontier = Some("bucket/child".to_string());
"covered_namespace"
} else {
let hash = hash_path("bucket/child");
let cycle = (0..data_usage_update_dir_cycles())
.find(|cycle| !hash.mod_(*cycle, data_usage_update_dir_cycles()))
.expect("a cycle samples other work");
scanner.old_cache.info.next_cycle = u64::from(cycle);
"sampled_namespace"
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut root,
)
.await
.expect("cached namespace reuses accounting");
let summary = scanner.new_cache.size_recursive("bucket").expect("complete bucket totals");
assert_eq!((summary.size, summary.objects), (17, 5));
let after = global_metrics().scan_observation().snapshot();
for label in [
"clean_prefix",
"covered_namespace",
"sampled_namespace",
"classified_namespace",
] {
assert_eq!(
after.cache_reuse[label].count,
before.cache_reuse[label].count + u64::from(label == kind),
"reuse reason {label}"
);
}
assert_eq!(after.metadata_reads.attempts, before.metadata_reads.attempts);
let gate = if covered { "covered_namespace" } else { "namespace_sampling" };
assert_eq!(after.reuse_decisions[gate].eligible, before.reuse_decisions[gate].eligible + 1);
}
}
async fn scan_hot_cold_segment_fixture(scoped: bool) -> (DataUsageEntry, Vec<String>) {
let (mut scanner, temp_dir) = build_test_scanner().await;
let visits = metadata_visits::observe(&mut scanner);
@@ -3465,7 +3278,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
parent: None,
object_heal_prob_div: 1,
};
let before = global_metrics().scan_observation().snapshot();
let mut root = DataUsageEntry::default();
scanner
.scan_folder(CancellationToken::new(), folder, &mut root)
@@ -3477,9 +3289,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
.size_recursive("bucket/clean")
.expect("erasure scan should visit the clean entry");
assert_eq!((clean.size, clean.objects), (0, 0));
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(observed.erasure_clean_prefix_walks, before.erasure_clean_prefix_walks + 1);
assert_eq!(observed.cache_reuse["clean_prefix"].count, before.cache_reuse["clean_prefix"].count);
}
#[tokio::test]
@@ -774,7 +774,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -791,15 +790,6 @@ async fn erasure_candidate_covered_namespace_reuses_totals_after_classification(
assert_eq!(budget.progress(), (0, 1), "only the parent consumes namespace quota");
assert_eq!(charged_ops(&root), 4, "list both directories once without re-reading covered metadata");
assert_eq!(scanner.new_cache.find(&folder.name).expect("reused candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
@@ -879,7 +869,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
},
);
scanner.budget = budget.clone();
let before = global_metrics().scan_observation().snapshot();
scanner
.scan_folder(
budget.token(),
@@ -900,15 +889,6 @@ async fn erasure_candidate_preserves_compacted_namespace_sampling_after_classifi
"sampling still requires full current classification, without metadata re-read"
);
assert_eq!(scanner.new_cache.find(&folder.name).expect("sampled candidate").objects, 1);
let observed = global_metrics().scan_observation().snapshot();
assert_eq!(
observed.cache_reuse["classified_namespace"].count,
before.cache_reuse["classified_namespace"].count + 1
);
assert_eq!(
observed.metadata_reads.attempts, before.metadata_reads.attempts,
"classification retains the existing zero metadata read path"
);
}
#[tokio::test]
+46 -133
View File
@@ -40,7 +40,6 @@ use rustfs_lock::{LockError, NamespaceLockGuard};
use rustfs_scanner_metrics::metrics::{
Metric, Metrics, emit_scan_bucket_drive_complete, emit_scan_bucket_drive_partial, global_metrics,
};
use rustfs_scanner_metrics::scan_observation::{PREFIX_REUSE_BLOCKERS, ScannerBaselineRejection, ScannerPrefixReuseStage};
use rustfs_utils::path::path_join_buf;
use sha2::{Digest as _, Sha256};
use std::collections::{BTreeMap, BTreeSet, HashMap, HashSet};
@@ -113,7 +112,16 @@ pub(crate) const SCANNER_SEGMENT_ACTIVATION_PROOF_INPUTS: [&str; 9] = [
"durable_dirty_producer_journal",
"ack_generation_guard",
];
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = PREFIX_REUSE_BLOCKERS;
pub(crate) const SCANNER_SEGMENT_ACTIVATION_FAIL_CLOSED_CHECKS: [&str; 8] = [
"missing_producer_identity",
"missing_durable_journal_replay",
"restart_gap",
"generation_gap",
"overflow",
"stale_ack_generation",
"missing_cold_zero_walk_oracle",
"distributed_without_peer_invalidation",
];
pub type DirtyUsageBuckets = HashMap<String, u64>;
@@ -189,7 +197,6 @@ struct VerifiedRemoteDirtyUsage {
}
struct ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Option<ScannerSegmentReuseActivationPreflight>,
scope: ScannerBucketScanScope,
remote_dirty_usage_acknowledgements: Vec<crate::scanner::ScannerDirtyUsageAcknowledgement>,
distributed_segment_invalidation_evidence: Option<DistributedSegmentInvalidationEvidence>,
@@ -229,13 +236,6 @@ pub(crate) struct ScannerSegmentReuseActivationPreflight {
}
impl ScannerSegmentReuseActivationPreflight {
fn record_observation(&self, stage: ScannerPrefixReuseStage) {
global_metrics().scan_observation().record_prefix_reuse(
stage,
self.scanner_segment_reuse_activated,
&self.fail_closed_blockers,
);
}
#[cfg(test)]
pub(crate) fn fail_closed_blockers(&self) -> impl Iterator<Item = &'static str> + '_ {
self.fail_closed_blockers.iter().filter_map(|blocker| *blocker)
@@ -319,7 +319,6 @@ fn resolve_remote_dirty_usage_scope(
baseline_proof: ScannerCacheBaselineProof<'_>,
) -> ScannerBucketScopeResolutionResult {
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -411,7 +410,6 @@ fn resolve_remote_dirty_usage_scope(
};
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
scope,
remote_dirty_usage_acknowledgements: scoped_acknowledgements,
distributed_segment_invalidation_evidence,
@@ -429,81 +427,43 @@ fn complete_scanner_cache_snapshot_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> Option<DataUsageScanPlanDigest> {
validate_scanner_cache_snapshot_plan_digest(snapshot, proof, expected_converged).ok()
}
fn validate_scanner_cache_snapshot_plan_digest(
snapshot: &DataUsageInfo,
proof: ScannerCacheBaselineProof<'_>,
expected_converged: Option<bool>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineRejection> {
// The observational mark disqualifies only the authoritative slot: an
// observed candidate carries it by construction.
let marks_accepted = match expected_converged {
Some(expected) => snapshot.usage_snapshot_converged == Some(expected),
None => snapshot.usage_snapshot_converged.is_some() && !snapshot.usage_snapshot_observational,
};
if !snapshot.is_complete_bucket_usage_snapshot() {
return Err(ScannerBaselineRejection::Incomplete);
}
if snapshot.usage_snapshot_partial {
return Err(ScannerBaselineRejection::Partial);
}
if !marks_accepted {
return Err(ScannerBaselineRejection::Marks);
}
if snapshot.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::Epoch);
}
if snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len() {
return Err(ScannerBaselineRejection::SourceCount);
if !snapshot.is_complete_bucket_usage_snapshot()
|| snapshot.usage_snapshot_partial
|| !marks_accepted
|| snapshot.scanner_epoch != Some(proof.leader_epoch)
|| snapshot.usage_snapshot_set_states.len() != proof.expected_sources.len()
{
return None;
}
// Completed maintenance also covers ordinary usage. Keep its exact stored
// proof for cache reuse, and reject mixtures of different set work proofs.
let baseline_plan_digest = DataUsageScanPlanDigest(
snapshot
.usage_snapshot_set_states
.first()
.and_then(|state| state.scan_plan_digest)
.ok_or(ScannerBaselineRejection::PlanMissing)?,
);
let baseline_plan_digest = DataUsageScanPlanDigest(snapshot.usage_snapshot_set_states.first()?.scan_plan_digest?);
if !scanner_plan_digest_is_work_variant(proof.scan_plan_digest, baseline_plan_digest) {
return Err(ScannerBaselineRejection::PlanMismatch);
return None;
}
let mut states = HashSet::with_capacity(snapshot.usage_snapshot_set_states.len());
for state in &snapshot.usage_snapshot_set_states {
let source = DataUsageCacheSource::new(
usize::try_from(state.pool_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
usize::try_from(state.set_index).map_err(|_| ScannerBaselineRejection::SourceIndex)?,
);
if !proof.expected_sources.contains(&source) {
return Err(ScannerBaselineRejection::SourceMembership);
}
if !states.insert(source) {
return Err(ScannerBaselineRejection::SourceDuplicate);
}
if !state.complete {
return Err(ScannerBaselineRejection::SetIncomplete);
}
if state.tombstone {
return Err(ScannerBaselineRejection::SetTombstone);
}
if state.scanner_epoch != Some(proof.leader_epoch) {
return Err(ScannerBaselineRejection::SetEpoch);
}
if state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle) {
return Err(ScannerBaselineRejection::SetCycle);
}
if state.scan_plan_digest != Some(baseline_plan_digest.0) {
return Err(ScannerBaselineRejection::SetPlan);
let source = DataUsageCacheSource::new(usize::try_from(state.pool_index).ok()?, usize::try_from(state.set_index).ok()?);
if !proof.expected_sources.contains(&source)
|| !states.insert(source)
|| !state.complete
|| state.tombstone
|| state.scanner_epoch != Some(proof.leader_epoch)
|| state.scanner_cycle.is_none_or(|cycle| cycle > proof.want_cycle)
|| state.scan_plan_digest != Some(baseline_plan_digest.0)
{
return None;
}
}
if states != *proof.expected_sources {
return Err(ScannerBaselineRejection::SourceCoverage);
}
Ok(baseline_plan_digest)
(states == *proof.expected_sources).then_some(baseline_plan_digest)
}
/// Whether `digest` is `structural` under any work flag: the same bucket
@@ -533,35 +493,10 @@ fn scanner_baseline_pending_buckets(proof: &ScannerCacheBaselineProof<'_>) -> Ve
}
fn complete_scanner_cache_baseline_plan_digest(proof: ScannerCacheBaselineProof<'_>) -> Option<DataUsageScanPlanDigest> {
validate_scanner_cache_baseline_plan_digest(proof).ok()
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
struct ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection,
observed: Option<ScannerBaselineRejection>,
}
fn validate_scanner_cache_baseline_plan_digest(
proof: ScannerCacheBaselineProof<'_>,
) -> std::result::Result<DataUsageScanPlanDigest, ScannerBaselineValidationFailure> {
let authoritative_error = |authoritative| ScannerBaselineValidationFailure {
authoritative,
observed: None,
};
let authoritative_data = proof
.authoritative_data
.ok_or_else(|| authoritative_error(ScannerBaselineRejection::Missing))?;
let authoritative = serde_json::from_slice::<DataUsageInfo>(authoritative_data)
.map_err(|_| authoritative_error(ScannerBaselineRejection::Decode))?;
let authoritative_rejection = match validate_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
Ok(validated_digest) => return Ok(validated_digest),
Err(reason) => reason,
};
let observed_error = |observed| ScannerBaselineValidationFailure {
authoritative: authoritative_rejection,
observed: Some(observed),
};
let authoritative = serde_json::from_slice::<DataUsageInfo>(proof.authoritative_data?).ok()?;
if let Some(validated_digest) = complete_scanner_cache_snapshot_plan_digest(&authoritative, proof, None) {
return Some(validated_digest);
}
// A complete but superseded observation may reuse its per-set cache only
// when it was explicitly tied to the durable authoritative baseline. It
@@ -571,18 +506,14 @@ fn validate_scanner_cache_baseline_plan_digest(
&& !authoritative.usage_snapshot_observational)
|| crate::scanner::data_usage_info_is_bootstrap_pending(&authoritative);
if !authoritative_has_identity {
return Err(observed_error(ScannerBaselineRejection::AuthoritativeIdentity));
return None;
}
let observed_data = proof
.observed_candidate_data
.ok_or_else(|| observed_error(ScannerBaselineRejection::Missing))?;
let observed =
serde_json::from_slice::<DataUsageInfo>(observed_data).map_err(|_| observed_error(ScannerBaselineRejection::Decode))?;
let observed = serde_json::from_slice::<DataUsageInfo>(proof.observed_candidate_data?).ok()?;
if !observed_data_usage_is_newer(&observed, &authoritative) {
return Err(observed_error(ScannerBaselineRejection::NotNewer));
return None;
}
validate_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false)).map_err(observed_error)
complete_scanner_cache_snapshot_plan_digest(&observed, proof, Some(false))
}
fn scanner_segment_reuse_baseline_producer_evidence(
@@ -727,12 +658,6 @@ impl ScannerScopeFallbackReason {
}
}
#[derive(Clone, Copy)]
struct ScannerScopeFallback {
reason: ScannerScopeFallbackReason,
baseline_rejection: Option<ScannerBaselineValidationFailure>,
}
/// Classify a resolved full-namespace scope against the preconditions
/// `resolve_scanner_bucket_scan_scope` checks, in the same order. Returns
/// `None` when the cycle was required to walk everything or was scoped.
@@ -743,24 +668,18 @@ fn scanner_scope_fallback_reason(
all_buckets: &[BucketInfo],
baseline_proof: ScannerCacheBaselineProof<'_>,
distributed: bool,
) -> Option<ScannerScopeFallback> {
let fallback = |reason| {
Some(ScannerScopeFallback {
reason,
baseline_rejection: None,
})
};
) -> Option<ScannerScopeFallbackReason> {
if !scope_eligible || !resolved_scope.is_default() {
return None;
}
if dirty_usage_snapshot.generation == u64::MAX {
return fallback(ScannerScopeFallbackReason::DirtyGenerationExhausted);
return Some(ScannerScopeFallbackReason::DirtyGenerationExhausted);
}
if !dirty_usage_snapshot.covers_listed_pending {
return fallback(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
return Some(ScannerScopeFallbackReason::DirtySnapshotIncomplete);
}
if dirty_usage_snapshot.buckets.len() > crate::SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES {
return fallback(ScannerScopeFallbackReason::DirtyBucketOverflow);
return Some(ScannerScopeFallbackReason::DirtyBucketOverflow);
}
// Peers may contribute dirty buckets the local snapshot does not hold.
if !distributed {
@@ -770,19 +689,13 @@ fn scanner_scope_fallback_reason(
.keys()
.any(|bucket| listed.contains(bucket.as_str()))
{
return fallback(ScannerScopeFallbackReason::NoListedDirtyBuckets);
return Some(ScannerScopeFallbackReason::NoListedDirtyBuckets);
}
}
if let Err(failure) = validate_scanner_cache_baseline_plan_digest(baseline_proof) {
return Some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::BaselineUnavailable,
baseline_rejection: Some(failure),
});
if complete_scanner_cache_baseline_plan_digest(baseline_proof).is_none() {
return Some(ScannerScopeFallbackReason::BaselineUnavailable);
}
distributed.then_some(ScannerScopeFallback {
reason: ScannerScopeFallbackReason::PeerScopeUnavailable,
baseline_rejection: None,
})
distributed.then_some(ScannerScopeFallbackReason::PeerScopeUnavailable)
}
fn scanner_segment_reuse_activation_preflight() -> ScannerSegmentReuseActivationPreflight {
+7 -11
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use super::*;
/// scanner cache locks and the cache snapshot persist/publish path.
use crate::data_usage_define::{DataUsageCacheBaselineRevisions, clone_cache_value};
use crate::data_usage_define::clone_cache_value;
use crate::{ScannerConfigObjectDelete, ScannerObjectIO};
use rustfs_scanner_metrics::usage_candidate::{
ScannerUsageCandidateKind as CandidateKind, ScannerUsageCandidateOutcome as CandidateOutcome,
@@ -1494,7 +1494,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
store: Arc<SetDisks>,
updates: &mpsc::Sender<DataUsageCache>,
mut cache_snapshot: DataUsageCache,
initial_revisions: Option<&DataUsageCacheBaselineRevisions>,
initial_revisions: Option<&DataUsageCacheRevisions>,
cache_cycle_floor: &AtomicU64,
expected_publication_epoch: u64,
) -> Option<SystemTime> {
@@ -1536,10 +1536,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
};
let mut persisted = DataUsageCache::default();
let current_baseline = match persisted
.load_with_baseline_revisions(store.clone(), DATA_USAGE_CACHE_NAME)
.await
{
let revisions = match persisted.load_with_revisions(store.clone(), DATA_USAGE_CACHE_NAME).await {
Ok(revisions) => revisions,
Err(err) => {
error!(
@@ -1561,7 +1558,6 @@ pub(super) async fn persist_and_publish_cache_snapshot(
return None;
}
};
let revisions = current_baseline.revisions();
let Some(scan_plan_digest) = cache_snapshot.info.scan_plan_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "plan_digest");
return None;
@@ -1625,9 +1621,9 @@ pub(super) async fn persist_and_publish_cache_snapshot(
{
cache_snapshot = persisted;
} else {
// Another execution must not advance the baseline. A backup catching
// up to the same decoded primary changes no baseline content.
if !initial_revisions.is_some_and(|initial| current_baseline.permits_unchanged_baseline_since(initial)) {
// A later execution may have completed while this scan was walking.
// Only replace the cache revision from which this scan started.
if initial_revisions != Some(&revisions) {
warn!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
@@ -1666,7 +1662,7 @@ pub(super) async fn persist_and_publish_cache_snapshot(
let done_save = Metrics::time(Metric::SaveUsage);
if let Err(e) = cache_snapshot
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, revisions, expected_publication_epoch)
.save_with_revisions_for_epoch(store.clone(), DATA_USAGE_CACHE_NAME, &revisions, expected_publication_epoch)
.await
{
error!(
+3 -48
View File
@@ -14,7 +14,6 @@
use super::*;
/// ScannerIOCache implementation for SetDisks: bucket ordering, worker fan-out, merge, and publish.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::{ScannerBucketReuseKind, ScannerScanObservation};
pub(super) async fn collect_completed_bucket_results(
ctx: CancellationToken,
@@ -143,32 +142,6 @@ pub(super) struct PreparedScopedSetScan {
pub(super) buckets: Vec<BucketInfo>,
pub(super) cache: DataUsageCache,
pub(super) cold_bucket_reuse_proof: Option<ScopedColdBucketReuseProof>,
cold_bucket_reuse_count: u64,
cold_bucket_reuse_elapsed: Duration,
}
impl PreparedScopedSetScan {
pub(super) fn record_cold_bucket_reuse(&self, observation: &ScannerScanObservation) {
observation.record_bucket_reuse(
ScannerBucketReuseKind::ScopedColdBucket,
self.cold_bucket_reuse_count,
self.cold_bucket_reuse_elapsed,
);
}
}
/// Counts only roots delivered to the collector; cancellation or channel failure is not reuse.
pub(super) async fn send_current_cache_root_entry(
bucket_result_tx: &mpsc::Sender<DataUsageEntryInfo>,
root: DataUsageEntryInfo,
cache: &DataUsageCache,
pending_maintenance_work: &AtomicBool,
observation: &ScannerScanObservation,
) -> std::result::Result<(), ScannerError> {
let started = Instant::now();
send_cache_root_entry(bucket_result_tx, root, cache, pending_maintenance_work).await?;
observation.record_bucket_reuse(ScannerBucketReuseKind::CurrentRoot, 1, started.elapsed());
Ok(())
}
pub(super) struct ScopedColdBucketReuseProof {
@@ -252,8 +225,6 @@ pub(super) fn prepare_scoped_set_scan(
};
cache.replace(DATA_USAGE_ROOT, "", DataUsageEntry::default());
let mut current_bucket_names = HashSet::with_capacity(all_buckets.len());
let mut cold_bucket_reuse_count = 0u64;
let mut cold_bucket_reuse_elapsed = Duration::ZERO;
for bucket in all_buckets {
if !current_bucket_names.insert(bucket.name.as_str()) {
return None;
@@ -262,14 +233,11 @@ pub(super) fn prepare_scoped_set_scan(
cache.info.scan_bucket_value_origins.remove(&bucket.name);
cache.replace(&bucket.name, DATA_USAGE_ROOT, DataUsageEntry::default());
} else {
let started = Instant::now();
cache.copy_with_children(
old_cache,
&rustfs_data_usage::hash_path(&bucket.name),
&Some(rustfs_data_usage::hash_path(DATA_USAGE_ROOT)),
);
cold_bucket_reuse_count = cold_bucket_reuse_count.saturating_add(1);
cold_bucket_reuse_elapsed = cold_bucket_reuse_elapsed.saturating_add(started.elapsed());
}
}
@@ -281,8 +249,6 @@ pub(super) fn prepare_scoped_set_scan(
.collect(),
cache,
cold_bucket_reuse_proof,
cold_bucket_reuse_count,
cold_bucket_reuse_elapsed,
})
}
@@ -459,10 +425,7 @@ impl ScannerIOCache for SetDisks {
.ok_or_else(|| StorageError::other("scanner cache publication is blocked by data movement"))?,
};
let mut old_cache = DataUsageCache::default();
let initial_revisions = match old_cache
.load_with_baseline_revisions(self.clone(), DATA_USAGE_CACHE_NAME)
.await
{
let initial_revisions = match old_cache.load_with_revisions(self.clone(), DATA_USAGE_CACHE_NAME).await {
Ok(revisions) => Some(revisions),
Err(e) => {
warn!(
@@ -592,8 +555,6 @@ impl ScannerIOCache for SetDisks {
})
});
let mut scoped_cache = scoped_scan.map(|mut prepared| {
// The copies are completed work even if this turn later fails publication.
prepared.record_cold_bucket_reuse(global_metrics().scan_observation());
buckets = prepared.buckets;
prepared.cache.info.scan_coverage_digest = Some(bucket_coverage_digest);
prepared.cache
@@ -1597,14 +1558,8 @@ impl ScannerIOCache for SetDisks {
}
record_completed_usage_watch(&usage_watch_completions_clone, source, &bucket.name, bucket_usage_watch.as_ref()).await;
if let Err(e) =
send_current_cache_root_entry(
&bucket_result_tx_clone,
*root,
&cache,
&pending_maintenance_work_clone,
global_metrics().scan_observation(),
)
.await
send_cache_root_entry(&bucket_result_tx_clone, *root, &cache, &pending_maintenance_work_clone)
.await
{
record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await;
error!(
+3 -20
View File
@@ -357,7 +357,6 @@ where
S: ScannerStorage,
{
let default_result = |scope: ScannerBucketScanScope| ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: None,
scope,
remote_dirty_usage_acknowledgements: Vec::new(),
distributed_segment_invalidation_evidence: None,
@@ -507,7 +506,7 @@ where
distributed,
resolution.baseline_proof,
);
let scope = scoped_scan_scope_from_dirty_buckets(
default_result(scoped_scan_scope_from_dirty_buckets(
resolution.requested_scope,
dirty_buckets,
(!distributed).then_some(resolution.dirty_usage_snapshot.scopes.as_ref()),
@@ -515,11 +514,7 @@ where
segment_reuse_activation_preflight.scanner_segment_reuse_activated,
resolution.all_buckets,
resolution.baseline_proof,
);
ScannerBucketScopeResolutionResult {
segment_reuse_activation_preflight: Some(segment_reuse_activation_preflight),
..default_result(scope)
}
))
}
/// Runs once between the bucket listing and the dirty-usage snapshot, the
@@ -764,9 +759,6 @@ where
},
)
.await;
if let Some(preflight) = scope_resolution.segment_reuse_activation_preflight {
preflight.record_observation(ScannerPrefixReuseStage::ScopeResolution);
}
let remote_dirty_usage_acknowledgements = scope_resolution.remote_dirty_usage_acknowledgements;
let distributed_segment_invalidation_evidence = scope_resolution.distributed_segment_invalidation_evidence;
let scan_scope = scope_resolution.scope;
@@ -778,14 +770,8 @@ where
baseline_proof,
distributed,
);
if let Some(fallback) = scope_fallback {
let reason = fallback.reason;
if let Some(reason) = scope_fallback {
global_metrics().record_scanner_scope_fallback(reason.as_str());
if let Some(failure) = fallback.baseline_rejection {
global_metrics()
.scan_observation()
.record_baseline_rejection(failure.authoritative, failure.observed);
}
tracing::info!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_SET_STATE,
@@ -799,7 +785,6 @@ where
"Scanner cycle walks every bucket because its dirty-bucket scope was not admitted"
);
}
let scope_fallback = scope_fallback.map(|fallback| fallback.reason);
let segment_invalidation_proof = dirty_usage_producer_evidence(&dirty_usage_snapshot).segment_invalidation_proof();
#[cfg(test)]
if let Some(observer) = resolved_scope_observer {
@@ -876,7 +861,6 @@ where
None,
false,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
return Ok(ScannerCycleResult::new(status, dirty_usage_clear)
.with_publication_epoch(publication_epoch)
.with_activity_digest(activity_digest)
@@ -1223,7 +1207,6 @@ where
distributed_segment_invalidation_evidence,
cold_zero_walk_oracle,
);
segment_reuse_activation_preflight.record_observation(ScannerPrefixReuseStage::CycleCompletion);
let pending_maintenance_work = pending_maintenance_work_for_cycle(&pending_maintenance_work, &results);
let observed_cycle_floor = cache_cycle_floor.load(Ordering::Acquire);
let required_cycle_floor = (observed_cycle_floor > want_cycle).then_some(observed_cycle_floor);
+2 -15
View File
@@ -15,7 +15,6 @@ use super::*;
use crate::UNKNOWN_TIER;
/// ScannerIODisk implementation for Disk: get_size and the per-disk bucket scan.
use crate::data_usage_define::clone_cache_value;
use rustfs_scanner_metrics::scan_observation::ScannerMetadataReadOutcome;
///
/// Seed [`SizeSummary::tier_stats`] from the cached tier-name list.
@@ -63,20 +62,8 @@ impl ScannerIODisk for Disk {
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let metadata_object_path = item.object_path();
let metadata_result = crate::workload_admission::with_scanner_metadata_io(async {
let read = global_metrics().start_scanner_metadata_read();
let result = self.read_metadata(&item.bucket, &metadata_object_path).await;
let (outcome, bytes) = match &result {
Ok(data) => (ScannerMetadataReadOutcome::Success, u64::try_from(data.len()).unwrap_or(u64::MAX)),
Err(e) if DiskError::is_err_object_not_found(e) || DiskError::is_err_version_not_found(e) => {
(ScannerMetadataReadOutcome::NotFound, 0)
}
Err(_) => (ScannerMetadataReadOutcome::Error, 0),
};
read.finish(outcome, bytes);
result
})
.await;
let metadata_result =
crate::workload_admission::with_scanner_metadata_io(self.read_metadata(&item.bucket, &metadata_object_path)).await;
ensure_scanner_node_operation_active(cancellation.as_ref())?;
let data = match metadata_result {
Ok(data) => data,
+7 -558
View File
@@ -27,10 +27,10 @@ use crate::storage_api::owner::{
};
use crate::storage_api::scan::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions, ObjectIO as _};
use crate::{
BUCKET_META_PREFIX, DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints,
InstanceContext, PoolEndpoints, ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER,
init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx,
new_disk, path2_bucket_object_with_base_path, save_config,
DATA_USAGE_BLOOM_NAME_PATH, DiskOption, ECStore, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
ScannerObjectOptions, ScannerPutObjReader, UNKNOWN_TIER, init_bucket_metadata_sys_for_scanner_tests,
init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
save_config,
};
use rustfs_concurrency::{
AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
@@ -104,7 +104,6 @@ fn scanner_activity_preflight_defers_a_temporarily_offline_peer() {
#[test]
fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
let before = global_metrics().scan_observation().snapshot();
let preflight = scanner_segment_reuse_activation_preflight();
assert!(preflight.production_activation);
@@ -125,9 +124,6 @@ fn scanner_segment_reuse_activation_preflight_reports_release_gate_inputs() {
"distributed_without_peer_invalidation",
]
);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated);
}
#[test]
@@ -2390,254 +2386,6 @@ async fn set_cache_send_failure_logs_receiver_and_snapshot_without_changing_retu
assert_set_cache_context(set_cache_event(&events, "publish_failed"), Some((0, 0)));
}
#[tokio::test]
#[serial]
async fn set_snapshot_accepts_backup_catchup_to_the_same_decoded_primary() {
use crate::data_usage_define::read_config_with_revision;
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read decoded primary bytes");
let primary_body = primary_body.expect("primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode older backup"))
.await
.expect("seed a lagging backup");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture primary with lagging backup");
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up without changing primary");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the reconciled backup");
assert_ne!(initial.revisions(), current.revisions());
assert!(current.permits_unchanged_baseline_since(&initial));
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
candidate.replace(
"photos",
DATA_USAGE_ROOT,
DataUsageEntry {
size: 20,
objects: 1,
..Default::default()
},
);
let cycle = candidate.info.next_cycle;
let cycle_floor = AtomicU64::new(cycle);
let (tx, mut rx) = mpsc::channel(1);
assert!(
persist_and_publish_cache_snapshot(Arc::clone(&set), &tx, candidate, Some(&initial), &cycle_floor, epoch)
.await
.is_some(),
"content-preserving backup catchup must not veto the new snapshot"
);
let published = rx.try_recv().expect("new snapshot reached the collector");
assert_eq!(published.info.next_cycle, cycle);
assert_eq!(published.find("photos").expect("new bucket entry").size, 20);
let (primary, primary_revision) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read new primary");
let (backup, backup_revision) = read_config_with_revision(set, &backup_path).await.expect("read new backup");
assert_eq!(primary, backup, "backup CAS must use the reconciled current revision");
assert_eq!(primary_revision, backup_revision);
let persisted = DataUsageCache::unmarshal(&primary.expect("new primary body")).expect("decode new primary");
assert_eq!(persisted.info.next_cycle, cycle);
assert_eq!(persisted.find("photos").expect("persisted new entry").size, 20);
}
#[tokio::test]
#[serial]
async fn set_snapshot_backup_catchup_keeps_revision_provenance_and_epoch_fences() {
use crate::data_usage_define::read_config_with_revision;
for case in ["different_backup", "different_primary", "corrupt_primary", "changed_epoch"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read original primary");
let primary_body = primary_body.expect("original primary exists");
let mut older = cache.clone();
older.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, older.marshal_msg().expect("encode lagging backup"))
.await
.expect("seed lagging backup");
if case == "corrupt_primary" {
save_config(Arc::clone(&set), &primary_path, b"corrupt-primary".to_vec())
.await
.expect("inject primary corruption before the initial load");
}
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture original baseline source and revisions");
let mut replacement = cache.clone();
replacement.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
match case {
"different_backup" => {
save_config(Arc::clone(&set), &backup_path, replacement.marshal_msg().expect("encode changed backup"))
.await
.expect("backup changes to different contents");
}
"different_primary" => {
let body = replacement.marshal_msg().expect("encode changed primary");
save_config(Arc::clone(&set), &primary_path, body.clone())
.await
.expect("primary advances");
save_config(Arc::clone(&set), &backup_path, body)
.await
.expect("backup follows changed primary");
}
"corrupt_primary" => {
save_config(Arc::clone(&set), &backup_path, b"corrupt-primary".to_vec())
.await
.expect("backup matches the corrupt primary ETag without a decoded primary proof");
}
"changed_epoch" => {
save_config(Arc::clone(&set), &backup_path, primary_body)
.await
.expect("backup catches up before epoch refusal");
}
_ => unreachable!("fixed test cases"),
}
let before_primary = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("current primary");
let before_backup = read_config_with_revision(Arc::clone(&set), &backup_path)
.await
.expect("current backup");
let mut candidate = cache;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let cycle_floor = AtomicU64::new(candidate.info.next_cycle);
let requested_epoch = if case == "changed_epoch" {
epoch.checked_add(1).expect("fixture epoch can advance")
} else {
epoch
};
let (tx, mut rx) = mpsc::channel(1);
let (result, events) = capture_set_cache_logs(persist_and_publish_cache_snapshot(
Arc::clone(&set),
&tx,
candidate,
Some(&initial),
&cycle_floor,
requested_epoch,
))
.await;
assert!(result.is_none(), "{case} must still refuse publication");
assert!(
matches!(rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"{case} forwards no snapshot"
);
let expected_state = if case == "changed_epoch" {
"save_failed"
} else {
"scan_baseline_revision_changed"
};
assert_set_cache_context(set_cache_event(&events, expected_state), Some((0, 0)));
assert_eq!(
read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("protected primary"),
before_primary,
"{case} does not overwrite primary"
);
assert_eq!(
read_config_with_revision(set, &backup_path).await.expect("protected backup"),
before_backup,
"{case} does not overwrite backup"
);
}
}
#[tokio::test]
#[serial]
async fn set_cache_cas_preserves_updates_after_backup_catchup_validation() {
use crate::data_usage_define::read_config_with_revision;
for changed_copy in ["primary", "backup"] {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let primary_path = path_join_buf(&[BUCKET_META_PREFIX, DATA_USAGE_CACHE_NAME]);
let backup_path = format!("{primary_path}.bkp");
let (primary_body, _) = read_config_with_revision(Arc::clone(&set), &primary_path)
.await
.expect("read initial primary");
let mut lagging = cache.clone();
lagging.info.scan_execution_digest = Some(DataUsageScanPlanDigest([8; 32]));
save_config(Arc::clone(&set), &backup_path, lagging.marshal_msg().expect("lagging backup bytes"))
.await
.expect("seed backup lag");
let mut loaded = DataUsageCache::default();
let initial = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read initial baseline");
save_config(Arc::clone(&set), &backup_path, primary_body.expect("initial primary exists"))
.await
.expect("finish backup catchup");
let current = loaded
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("read reconciled baseline");
assert!(current.permits_unchanged_baseline_since(&initial));
let changed_path = if changed_copy == "primary" {
&primary_path
} else {
&backup_path
};
let mut concurrent = cache.clone();
concurrent.info.scan_execution_digest = Some(DataUsageScanPlanDigest([9; 32]));
let concurrent_body = concurrent.marshal_msg().expect("encode concurrent update");
save_config(Arc::clone(&set), changed_path, concurrent_body.clone())
.await
.expect("copy changes after the baseline validation");
let mut candidate = cache;
candidate.info.next_cycle += 1;
candidate.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let result = candidate
.save_with_revisions_for_epoch(Arc::clone(&set), DATA_USAGE_CACHE_NAME, current.revisions(), epoch)
.await;
if changed_copy == "primary" {
assert!(result.is_err(), "primary CAS refuses the intervening update");
} else {
result.expect("primary remains durable when the best-effort backup CAS refuses an update");
}
let (protected, _) = read_config_with_revision(Arc::clone(&set), changed_path)
.await
.expect("read protected concurrent copy");
assert_eq!(
protected,
Some(concurrent_body),
"{changed_copy} CAS must not overwrite the concurrent update"
);
let (primary, _) = read_config_with_revision(set, &primary_path)
.await
.expect("read resulting primary");
let primary = DataUsageCache::unmarshal(&primary.expect("primary exists")).expect("primary remains decodable");
assert_eq!(
primary.info.scan_execution_digest,
Some(DataUsageScanPlanDigest([if changed_copy == "primary" { 9 } else { 3 }; 32])),
);
}
}
#[tokio::test]
#[serial]
async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers() {
@@ -2652,7 +2400,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
.expect("seed legacy set cache");
let mut persisted = DataUsageCache::default();
let initial = persisted
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the shared starting revision");
let mut fresh = legacy.clone();
@@ -2678,7 +2426,7 @@ async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers
assert_eq!(published.find("photos").expect("published bucket").size, 20);
assert_eq!(published.info.scan_execution_digest, fresh.info.scan_execution_digest);
let current = persisted
.load_with_baseline_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.load_with_revisions(Arc::clone(&set), DATA_USAGE_CACHE_NAME)
.await
.expect("capture the current revision for the unidentified execution");
@@ -2969,130 +2717,6 @@ fn scoped_scan_requires_a_complete_authoritative_baseline_with_exact_set_provena
);
}
#[test]
fn scoped_scan_baseline_rejections_preserve_the_existing_first_failed_check() {
let source = DataUsageCacheSource::new(1, 2);
let expected_sources = HashSet::from([source]);
let digest = DataUsageScanPlanDigest([9; 32]);
let baseline = complete_usage_baseline(source, digest, 7, 11);
let snapshot: DataUsageInfo = serde_json::from_slice(&baseline).expect("decode complete baseline");
let proof = ScannerCacheBaselineProof {
authoritative_data: Some(&baseline),
observed_candidate_data: None,
expected_sources: &expected_sources,
leader_epoch: 11,
want_cycle: 8,
scan_plan_digest: digest,
};
type RejectionCase = (ScannerBaselineRejection, fn(&mut DataUsageInfo));
let cases: [RejectionCase; 13] = [
(ScannerBaselineRejection::Incomplete, |s| s.usage_snapshot_complete = false),
(ScannerBaselineRejection::Partial, |s| s.usage_snapshot_partial = true),
(ScannerBaselineRejection::Marks, |s| s.usage_snapshot_converged = None),
(ScannerBaselineRejection::Epoch, |s| s.scanner_epoch = Some(10)),
(ScannerBaselineRejection::SourceCount, |s| s.usage_snapshot_set_states.clear()),
(ScannerBaselineRejection::PlanMissing, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = None
}),
(ScannerBaselineRejection::PlanMismatch, |s| {
s.usage_snapshot_set_states[0].scan_plan_digest = Some([8; 32])
}),
(ScannerBaselineRejection::SourceMembership, |s| {
s.usage_snapshot_set_states[0].set_index = 3
}),
(ScannerBaselineRejection::SetIncomplete, |s| {
s.usage_snapshot_set_states[0].complete = false
}),
(ScannerBaselineRejection::SetTombstone, |s| {
s.usage_snapshot_set_states[0].tombstone = true
}),
(ScannerBaselineRejection::SetEpoch, |s| {
s.usage_snapshot_set_states[0].scanner_epoch = None
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = Some(9)
}),
(ScannerBaselineRejection::SetCycle, |s| {
s.usage_snapshot_set_states[0].scanner_cycle = None
}),
];
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&snapshot, proof, None), Ok(digest));
for (expected, mutate) in cases {
let mut invalid = snapshot.clone();
mutate(&mut invalid);
assert_eq!(validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None), Err(expected));
assert_eq!(complete_scanner_cache_snapshot_plan_digest(&invalid, proof, None), None);
}
let mut invalid = snapshot.clone();
invalid.usage_snapshot_partial = true;
invalid.scanner_epoch = Some(10);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&invalid, proof, None),
Err(ScannerBaselineRejection::Partial)
);
let two_sources = HashSet::from([source, DataUsageCacheSource::new(1, 3)]);
let mut two_sets = snapshot.clone();
two_sets
.usage_snapshot_set_states
.push(two_sets.usage_snapshot_set_states[0].clone());
let two_set_proof = ScannerCacheBaselineProof {
expected_sources: &two_sources,
..proof
};
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SourceDuplicate)
);
two_sets.usage_snapshot_set_states[1].set_index = 3;
two_sets.usage_snapshot_set_states[1].scan_plan_digest = Some([8; 32]);
assert_eq!(
validate_scanner_cache_snapshot_plan_digest(&two_sets, two_set_proof, None),
Err(ScannerBaselineRejection::SetPlan)
);
let undecodable = Bytes::from_static(b"not-json");
for (data, expected) in [
(None, ScannerBaselineRejection::Missing),
(Some(&undecodable), ScannerBaselineRejection::Decode),
] {
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
authoritative_data: data,
..proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: expected,
observed: None
})
);
}
let mut wrong_epoch = snapshot;
wrong_epoch.scanner_epoch = Some(10);
let wrong_epoch = Bytes::from(serde_json::to_vec(&wrong_epoch).expect("encode stale epoch baseline"));
let stale_proof = ScannerCacheBaselineProof {
authoritative_data: Some(&wrong_epoch),
..proof
};
assert_eq!(
validate_scanner_cache_baseline_plan_digest(stale_proof),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Missing),
})
);
assert_eq!(
validate_scanner_cache_baseline_plan_digest(ScannerCacheBaselineProof {
observed_candidate_data: Some(&undecodable),
..stale_proof
}),
Err(ScannerBaselineValidationFailure {
authoritative: ScannerBaselineRejection::Epoch,
observed: Some(ScannerBaselineRejection::Decode)
})
);
}
#[test]
fn scoped_scan_accepts_only_a_complete_observation_tied_to_the_authoritative_baseline() {
let source = DataUsageCacheSource::new(1, 2);
@@ -3526,12 +3150,6 @@ fn distributed_segment_reuse_activation_keeps_remote_dirty_buckets_at_bucket_sco
result.scope.prefix_scope_for("photos").is_some(),
"durable local producer proof may activate local segment reuse after distributed ACK capability evidence"
);
assert!(
result
.segment_reuse_activation_preflight
.expect("actual scope proof is retained for observation")
.scanner_segment_reuse_activated
);
assert!(
result.scope.prefix_scope_for("archive").is_none(),
"remote dirty usage has only bucket-granularity evidence and must not be narrowed to a local prefix"
@@ -3950,10 +3568,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
.checked_flatten("stable")
.expect("selected bucket placeholder should exist");
assert_eq!((stable.size, stable.objects), (0, 0));
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
assert!(prepared.cache.find("stable/prefix").is_none());
assert_eq!(prepared.cache.find("dirty").map(|entry| (entry.size, entry.objects)), Some((0, 0)));
assert!(prepared.cache.find("deleted").is_none());
@@ -3965,124 +3579,6 @@ fn scoped_set_scan_rebuilds_selected_buckets_and_drops_deleted_buckets() {
assert_eq!(prepared.cache.info.lkg_scan_plan_digest, Some(baseline_digest));
}
#[test]
fn cold_bucket_reuse_observation_rejects_unbound_or_failed_preparation() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
let mut old_cache = complete_set_usage_cache(&[("stable", 10), ("dirty", 20)], baseline_digest);
old_cache.info.scan_bucket_incarnations = test_bucket_incarnations(&["stable", "dirty"]);
let incarnations = old_cache.info.scan_bucket_incarnations.clone();
let buckets = vec![
bucket_info_with_created_time("stable"),
bucket_info_with_created_time("dirty"),
];
let scope = ScannerBucketScanScope {
selected_buckets: Some(Arc::new(HashSet::from(["dirty".to_string()]))),
selected_bucket_prefixes: None,
baseline_scan_plan_digest: Some(baseline_digest),
};
let generation = ScannerSetCacheGeneration {
want_cycle: 8,
leader_epoch: 11,
tier_registry_generation: 13,
source: DataUsageCacheSource::new(1, 2),
scan_plan_digest: DataUsageScanPlanDigest([2; 32]),
};
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
let prepared = prepare_scoped_set_scan(&old_cache, &buckets, &buckets, &scope, generation, Some(&incarnations))
.expect("bound complete cache permits a cold bucket copy");
prepared.record_cold_bucket_reuse(&observation);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 1);
for case in ["incomplete", "duplicate", "nil_incarnation"] {
let mut invalid_cache = old_cache.clone();
let mut invalid_buckets = buckets.clone();
let mut invalid_incarnations = incarnations.clone();
match case {
"incomplete" => invalid_cache.info.snapshot_complete = false,
"duplicate" => invalid_buckets.push(buckets[0].clone()),
"nil_incarnation" => {
invalid_incarnations.insert("stable".to_string(), Uuid::nil());
}
_ => unreachable!("fixed test cases"),
}
let prepared = prepare_scoped_set_scan(
&invalid_cache,
&invalid_buckets,
&invalid_buckets,
&scope,
generation,
Some(&invalid_incarnations),
);
assert!(prepared.is_none(), "{case} must not return a reusable prepared cache");
if let Some(prepared) = prepared {
prepared.record_cold_bucket_reuse(&observation);
}
assert_eq!(
observation.snapshot().bucket_reuse["scoped_cold_bucket"].count,
1,
"{case} does not count completed reuse"
);
}
}
#[tokio::test]
async fn current_root_reuse_counts_only_successful_collector_delivery() {
use rustfs_scanner_metrics::scan_observation::ScannerScanObservation;
let observation = ScannerScanObservation::default();
let cache = DataUsageCache::default();
let pending = AtomicBool::new(false);
let root = || DataUsageEntryInfo {
name: "bucket".to_string(),
parent: DATA_USAGE_ROOT.to_string(),
entry: DataUsageEntry {
size: 10,
objects: 2,
..Default::default()
},
bucket_incarnation: Some(Uuid::from_u128(7)),
tier_registry_generation: None,
value_origin: None,
};
let (sender, mut receiver) = mpsc::channel(1);
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.expect("current root is delivered to the collector");
let delivered = receiver.recv().await.expect("collector received the root");
assert_eq!((delivered.entry.size, delivered.entry.objects), (10, 2));
assert_eq!(observation.snapshot().bucket_reuse["current_root"].count, 1);
drop(receiver);
assert!(
super::io_cache::send_current_cache_root_entry(&sender, root(), &cache, &pending, &observation)
.await
.is_err()
);
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"closed collector is not completed reuse"
);
let (sender, _receiver) = mpsc::channel(1);
sender.send(root()).await.expect("fill bounded collector channel");
{
let mut send = Box::pin(super::io_cache::send_current_cache_root_entry(
&sender,
root(),
&cache,
&pending,
&observation,
));
assert!(futures::poll!(send.as_mut()).is_pending(), "full collector keeps reuse unfinished");
}
assert_eq!(
observation.snapshot().bucket_reuse["current_root"].count,
1,
"dropping unfinished delivery is not reuse"
);
assert_eq!(observation.snapshot().bucket_reuse["scoped_cold_bucket"].count, 0);
}
#[test]
fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
let baseline_digest = DataUsageScanPlanDigest([1; 32]);
@@ -4124,12 +3620,6 @@ fn scoped_set_scan_reuses_unselected_buckets_with_matching_incarnations() {
)
.expect("matching bucket incarnations should authorize cold bucket reuse");
let observation = rustfs_scanner_metrics::scan_observation::ScannerScanObservation::default();
prepared.record_cold_bucket_reuse(&observation);
let observed = observation.snapshot();
assert_eq!(observed.bucket_reuse["scoped_cold_bucket"].count, 1);
assert_eq!(observed.bucket_reuse["current_root"].count, 0);
assert!(observed.cache_reuse.values().all(|reuse| reuse.count == 0));
assert_eq!(prepared.buckets.iter().map(|bucket| bucket.name.as_str()).collect::<Vec<_>>(), ["dirty"]);
let stable = prepared
.cache
@@ -5135,36 +4625,12 @@ async fn get_size_treats_missing_metadata_as_skip_file() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let err = disk
.get_size(item.clone())
.get_size(item)
.await
.expect_err("missing metadata should be skipped instead of reported as a scanner failure");
assert!(matches!(err, StorageError::Io(ref io) if io.to_string() == SCANNER_SKIP_FILE_ERROR));
let missing = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(missing.attempts, before.attempts + 1);
assert_eq!(missing.not_found, before.not_found + 1);
assert_eq!(missing.bytes, before.bytes);
#[cfg(unix)]
{
tokio::fs::create_dir_all(&object_dir)
.await
.expect("recreate object directory");
std::os::unix::fs::symlink(STORAGE_FORMAT_FILE, &metadata_path).expect("inject a metadata symlink loop");
let err = disk
.get_size(item)
.await
.expect_err("symlink loop must remain a read failure");
assert!(is_scanner_metadata_transient_error(&err), "read error: {err:?}");
let failed = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(failed.attempts, missing.attempts + 1);
assert_eq!(failed.errors, missing.errors + 1);
assert_eq!(failed.succeeded, before.succeeded);
assert_eq!(failed.bytes, before.bytes);
}
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -5215,7 +4681,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
for contents in [b"".as_slice(), b"not-valid-filemeta".as_slice()] {
tokio::fs::write(&metadata_path, contents)
.await
@@ -5227,16 +4692,6 @@ async fn get_size_marks_corrupt_metadata_for_heal() {
assert!(is_scanner_metadata_corrupt_error(&err));
}
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 2);
assert_eq!(
reads.succeeded,
before.succeeded + 2,
"storage reads succeed before decoding rejects corruption"
);
assert_eq!(reads.errors, before.errors);
assert_eq!(reads.bytes, before.bytes + 18);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -5309,17 +4764,11 @@ async fn get_size_counts_delete_markers_separately_from_versions() {
debug: false,
};
let before = global_metrics().scan_observation().snapshot().metadata_reads;
let expected_bytes = tokio::fs::metadata(&metadata_path).await.expect("stat stored metadata").len();
let summary = disk.get_size(item).await.expect("scanner should read versioned metadata");
assert_eq!(summary.versions, 2);
assert_eq!(summary.delete_markers, 1);
assert_eq!(summary.total_size, 30);
let reads = global_metrics().scan_observation().snapshot().metadata_reads;
assert_eq!(reads.attempts, before.attempts + 1);
assert_eq!(reads.succeeded, before.succeeded + 1);
assert_eq!(reads.bytes, before.bytes + expected_bytes);
let _ = tokio::fs::remove_dir_all(&temp_dir).await;
}
@@ -183,11 +183,9 @@ async fn dirty_entry_removed_during_listing_still_forces_a_full_scan() {
);
assert_eq!(second.buckets_usage["hot-bucket"].objects_count, 2);
assert_eq!(second.buckets_usage["cold-bucket"].objects_count, 2);
let report = global_metrics().report().await;
let freshness = report.usage_freshness;
let freshness = global_metrics().report().await.usage_freshness;
assert_eq!(freshness.scope_fallback_total, fallbacks_before + 1);
assert_eq!(freshness.last_scope_fallback_reason, "dirty_snapshot_incomplete");
assert_eq!(report.scan_observation.scope_fallbacks["dirty_snapshot_incomplete"], 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -201,53 +199,11 @@ async fn required_full_scan_is_not_reported_as_a_scope_fallback() {
wait_for_namespace_commit_tails(store.as_ref()).await;
super::io_cycle::set_bucket_listing_window_hook_for_tests(|| record_dirty_usage_bucket("created-after-listing"));
let before = global_metrics().scan_observation().snapshot();
let (result, _, scope) = run_cycle(&store, 2, Some(baseline), true).await;
assert!(scope.is_default());
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert_eq!(result.scope_fallback(), None, "scheduled maintenance walks are not degradations");
assert_eq!(result.completed_scope_fallback(), None);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(after.scope_fallbacks, before.scope_fallbacks);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
#[tokio::test]
#[serial]
async fn baseline_unavailable_cycle_records_real_scope_and_baseline_rejections_once() {
let (_temp_dir, store) = setup_local_scanner_store(1, 1).await;
let _baseline = converged_baseline(&store).await;
put_fixture_object(&store, "hot-bucket", "second").await;
record_dirty_usage_bucket("hot-bucket");
wait_for_namespace_commit_tails(store.as_ref()).await;
let before = global_metrics().scan_observation().snapshot();
let (result, snapshot, scope) = run_cycle(&store, 2, Some(Bytes::from_static(b"not-json")), false).await;
assert!(scope.is_default());
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(result.scope_fallback(), Some(ScannerScopeFallbackReason::BaselineUnavailable));
assert_eq!(snapshot.buckets_usage["hot-bucket"].objects_count, 2);
let after = global_metrics().scan_observation().snapshot();
assert_eq!(
after.scope_fallbacks["baseline_unavailable"],
before.scope_fallbacks["baseline_unavailable"] + 1
);
assert_eq!(
after.authoritative_baseline_rejections["decode"],
before.authoritative_baseline_rejections["decode"] + 1
);
assert_eq!(
after.observed_baseline_rejections, before.observed_baseline_rejections,
"an undecodable authority never admits an observation"
);
assert_eq!(after.prefix_scope_resolution.evaluated, before.prefix_scope_resolution.evaluated + 1);
assert_eq!(after.prefix_scope_resolution.activated, before.prefix_scope_resolution.activated);
assert_eq!(
after.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"],
before.prefix_scope_resolution.blockers["missing_cold_zero_walk_oracle"] + 1
);
assert_eq!(after.prefix_cycle_completion.evaluated, before.prefix_cycle_completion.evaluated + 1);
clear_dirty_usage_buckets_for_tests();
}
@@ -1,12 +1,12 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
//! A bucket whose walk stops early can publish its last certified value as
//! pending. This fixture tests usage-scan status, not the outer scanner cycle
//! outcome: budget exhaustion may leave complete validated usage while the
//! coordinator records `Partial`. A continuation under a newer plan exercises
//! `mixed_coverage` (rustfs/rustfs#8494). A bucket that never certified keeps
//! the usage scan incomplete.
//! A bucket whose walk stops early without exhausting the cycle budget (here
//! a sweep resumed under a newer plan, the `mixed_coverage` case of
//! rustfs/rustfs#8494) no longer makes the whole cycle `Incomplete`. The set
//! publishes the bucket's last certified value, the cycle completes with the
//! bucket pending, and the next cycle walks it again. A bucket that never
//! certified keeps the cycle incomplete.
use super::*;
@@ -38,7 +38,8 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
..Default::default()
},
);
// Real checkpoint commits need the runtime drain grace after the count budget stops the walk.
// These cycles persist real bucket checkpoints and source caches. Keep
// CI storage latency inside the production drain window after a count stop.
budget.use_production_drain_grace_for_tests();
let (updates, mut receiver) = mpsc::channel(1);
let result = tokio::time::timeout(
@@ -48,7 +49,7 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
ScannerCycleRequest {
owner_ctx: ctx.clone(),
ctx: budget.token(),
budget: Arc::clone(&budget),
budget,
updates,
want_cycle: cycle,
leader_epoch: LEADER_EPOCH,
@@ -68,13 +69,6 @@ async fn run_cycle(store: &Arc<ECStore>, cycle: u64, options: CycleOptions) -> (
.await
.expect("cycle should finish within the test deadline")
.expect("cycle should succeed");
if options.max_objects.is_some() {
assert_eq!(
budget.reason(),
Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects),
"a count-limited fixture must stop for its object budget"
);
}
let snapshot = receiver.recv().await;
assert!(receiver.recv().await.is_none(), "a cycle publishes at most one terminal snapshot");
(result, snapshot)
@@ -131,8 +125,9 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
assert!(result.clear_verified_usage().is_empty());
assert!(dirty_usage_buckets_for_tests().is_empty());
// Cycle 3: the budget stops the walk and saves a checkpoint. Usage can
// carry the certified value as pending; the outer coordinator records Partial.
// Cycle 3: the object budget stops the hot bucket's walk and leaves a
// checkpoint under this cycle's plan. Its certified value is carried, so
// budget exhaustion leaves the bucket pending without blocking the cycle.
for object in ["p4/third", "p5/third"] {
put_fixture_object(&store, "hot-bucket", object).await;
}
@@ -149,38 +144,12 @@ async fn partial_bucket_with_a_certified_value_completes_the_cycle_as_pending()
},
)
.await;
assert_eq!(
result.status,
ScannerCycleStatus::Complete,
"certified carried usage remains complete after a budget stop"
);
let third = third.expect("validated carried usage publishes after the budget stop");
let third = third.expect("a carried partial cycle publishes");
assert_eq!(result.status, ScannerCycleStatus::Complete);
assert_eq!(result.pending_dirty_usage_buckets(), 1);
assert!(!third.usage_snapshot_observational);
assert_eq!(third.usage_snapshot_converged, Some(false));
assert_eq!(third.usage_snapshot_pending_buckets, vec!["hot-bucket".to_string()]);
assert_eq!(third.usage_snapshot_pending_bucket_count, 1);
assert!(third.is_complete_bucket_usage_snapshot());
let carried_hot = &third.buckets_usage["hot-bucket"];
let certified_hot = &second.buckets_usage["hot-bucket"];
assert_eq!(carried_hot.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Carried));
assert!(certified_hot.usage_certification.is_some(), "cycle 2 certified the prior hot value");
assert_eq!(
carried_hot.usage_certification, certified_hot.usage_certification,
"carrying must not mint a new certificate"
);
assert_eq!(carried_hot.objects_count, 4);
assert_eq!(carried_hot.size, certified_hot.size);
assert_eq!(third.buckets_usage["cold-bucket"].objects_count, 1);
assert_eq!(third.buckets_usage["idle-bucket"].objects_count, 1);
assert_eq!(third.objects_total_count, 6);
// Only hot was dirty in this scoped scan; it received no fresh certification or ACK.
assert!(result.clear_verified_usage().is_empty());
assert_eq!(
dirty_usage_buckets_for_tests().keys().cloned().collect::<Vec<_>>(),
vec!["hot-bucket".to_string()],
"the budget-stopped hot bucket retains its dirty responsibility"
);
assert_eq!(third.buckets_usage["hot-bucket"].objects_count, 4);
let set = Arc::clone(&store.pools[0].disk_set[0]);
let mut interrupted = DataUsageCache::default();
interrupted
@@ -137,7 +137,7 @@ For batch-delete codec acceptance, run `delete_versions_request_encoding_preserv
```bash
cargo nextest run -p rustfs-ecstore --lib --features test-util --no-tests fail \
-E 'test(delete_versions_request_encoding_preserves_batches_and_fleet_guards)'
NO_PROXY='*' no_proxy='*' cargo nextest run -p rustfs --lib --features gateway-stack --no-tests fail \
NO_PROXY='*' no_proxy='*' cargo nextest run -p rustfs --lib --no-tests fail \
-E 'test(delete_versions_wire_roundtrip_preserves_json_and_binary_only_batches)'
```
@@ -898,42 +898,6 @@ done
The `aggregated.scanner` payload preserves the same scanner progress, checkpoint, pacing, source work, maintenance control, lifecycle expiry, and lifecycle transition fields used by the local scanner status, but only for the responding node; `by_host.*.scanner` keeps that node's host view. Compare the per-node artifacts externally to find old active paths, partial checkpoints, pacing pressure, source-level control pressure, or downstream queue admission problems across the deployment.
## Reading Cached Bucket Statistics
Admin usage reads cache the selected scanner baseline for the existing 30-second TTL, then verify every returned bucket against uncached storage-owned incarnation metadata. Identity reads use at most four concurrent operations and a shared two-second deadline after releasing snapshot/refresh locks. Each bucket read linearizes independently against metadata transactions; the response is not an atomic snapshot of the entire namespace. Delayed peer invalidation cannot authorize a predecessor's statistics for a recreated bucket.
Missing, failed, timed-out, nil, mismatched, or legacy absent identities leave the bucket value absent (unknown), rather than returning the predecessor's value or a confirmed zero. Legacy statistics become visible after a scanner snapshot supplies the current identity. Account-info and usage display apply the existing single-process memory overlay before the final identity filter; distributed local counters are not promoted. When a projection omits values or changes an overlay, totals are partial and `usage_snapshot_converged=false`, and per-tier/replication aggregates are unavailable. Verified cold bucket values remain visible. Such response-only projections cannot be persisted, used for quota admission, or acknowledge dirty work.
## Reading Scanner Work Observations
`GET /rustfs/admin/v3/scanner/status` exposes `metrics.scan_observation` for the responding node. Capture timestamped snapshots on every node, including the scanner leader, and compare their deltas over the same workload window. These process-lifetime counters reset on restart; a missing field on an older node means unobserved. They are loaded independently and may briefly disagree while operations finish. This telemetry does not change scan cadence, cached usage reads, quota authority, dirty acknowledgement, or maintenance requirements.
| Field | Meaning |
|---|---|
| `metadata_reads.attempts` | Disk metadata read futures started after scanner node admission. Invalid paths and operations cancelled before starting the read do not count. |
| `metadata_reads.succeeded`, `not_found`, `errors`, `cancelled` | Storage outcomes, including object/version disappearance and dropped read futures. Successful storage reads can still contain corrupt metadata; decoding and lifecycle failures do not become read errors. Cancellation observes the future, not proof that physical I/O stopped. |
| `metadata_reads.bytes` | Total returned buffer lengths for successful reads, including inline data and corrupt metadata. Failed reads contribute no known bytes; this is not physical disk traffic. |
| `metadata_reads.elapsed_nanos` | Sum of read-future elapsed time, including disk-permit or RPC waits, failures, and cancellation. Folder pacing, scanner node admission, decoding, and lifecycle work are outside this interval. Concurrent reads overlap, so the sum is not cycle wall time. |
| `cache_reuse.clean_prefix` | Clean root subtrees copied under an admitted prefix scope by the non-erasure scanner. |
| `cache_reuse.covered_namespace` | Namespace subtrees copied from validated checkpoint coverage. |
| `cache_reuse.sampled_namespace` | Cached namespace subtrees copied when the existing periodic sampling selected other work. |
| `cache_reuse.classified_namespace` | EC candidate namespace subtrees copied only after the current listing finished classifying them. Health/classification work still runs. |
| `cache_reuse.*.count`, `elapsed_nanos` | Completed subtree reuse decisions and time spent copying/attaching their cache entries. Counts are not objects skipped or a cache hit ratio; times exclude classification/listing and are not total cache maintenance cost. |
| `bucket_reuse.scoped_cold_bucket` | Whole cold buckets copied and attached to a successfully prepared scoped set cache. Count is copied buckets, not plans selected; later turn publication can still fail. Time covers copying/attachment only. |
| `bucket_reuse.current_root` | Existing current bucket roots successfully sent to the local collector after watch, namespace lease and publication epoch checks. Time covers channel send only; failure or cancellation contributes no completed reuse. Collector acceptance and durable publication remain separate. |
| `reuse_decisions.*.not_evaluated`, `blocked`, `eligible` | First conditions reached by production clean-prefix, checkpoint-coverage and namespace-sampling gates. Policy exclusions (`missing_scope`, `no_resume_sweep`, `deep_scan`, `forward_sweep`, `active_replication`) precede cache proof checks. `blocked` reports the first failed work/cache condition; later checks are not repeated for diagnostics. `eligible` does not certify a copy: an EC candidate can still require classification. |
| `erasure_clean_prefix_walks` | Clean root children that still require walking because EC scans perform independent object-health work. |
| `prefix_scope_resolution.evaluated`, `activated`, `blockers` | Proof evaluations retained by the final scope resolver, admitted proof gates, and every failed prerequisite. An admitted proof does not guarantee a reusable child exists; actual subtree copies appear in `cache_reuse`. Startup/required full scans, early scope rejection, and discarded peer/ACK resolutions do not count here. |
| `prefix_cycle_completion.evaluated`, `activated`, `blockers` | End-of-cycle proof evaluations for subsequent reuse. An activated completion proof does not mean this cycle reused prefixes or that its later publication/persistence succeeded. One evaluation can increment several blocker counters. |
| `scope_fallbacks` | Per-reason breakdown of the existing `usage_freshness.scope_fallback_total`; required full scans are excluded. |
| `authoritative_baseline_rejections`, `observed_baseline_rejections` | First failed validation checks for `baseline_unavailable` fallback cycles, preserving the authoritative rejection when the observed fallback also fails. Missing/undecodable authority never reaches observed validation; `authoritative_identity` means it could not anchor an observed fallback. |
`freshness` describes the responding node's completed-cycle age, not whether every bucket has incorporated new writes. Read `metrics.usage_freshness` for node-local live pending work, guardrail vetoes, durable success and publication state; read `metrics.usage_publication` for the confirmed snapshot's pending count and publication class/outcome, and `cycle_schedule` for effective retry intervals and consecutive superseded cycles. Missing or unknown publication data does not establish zero pending work. A follower or a single node's empty dirty map cannot establish cluster-wide convergence.
Whole cold bucket copies, current-root reuse and the four subtree categories have different prerequisites and are not interchangeable hit rates. EC clean prefixes retain health walks. Deep or required full work must run, but a completed cache matching that exact work digest may satisfy a retry of the same cycle. A normal workload need not exercise current-root reuse or every subtree category; use controlled tests for positive emission evidence and capture real gate reasons when a VM window has zero copies.
The existing `life_time_ops.read_metadata` and `last_minute.actions.read_metadata` now include finished and cancelled read attempts; the latter reports count, accumulated whole seconds, and returned bytes in its existing window. Use `metadata_reads.elapsed_nanos` for subsecond read costs. Use read-count and byte deltas to assess work growth, and elapsed-time deltas to assess read-future cost. Compare those with reused-subtree deltas, scope fallback reasons, cycle wall time, pacing waits, and source work before attributing freshness delays to metadata I/O or proposing a scan optimization.
## Reading Lifecycle Transition Status
`metrics.lifecycle_transition`:
+4 -6
View File
@@ -56,13 +56,11 @@ The four site-replication state-writer concurrency proofs reserve the available
| PR, non-doc change | `Typos` | `ci.yml` `typos` | Via aggregate | `typos` |
| Full selection | `Workspace tests` (serial ILM step) | `ci.yml` `workspace-tests` | Via aggregate | exact `Run ignored ILM integration tests serially` command in the job |
| Nightly schedule, manual dispatch | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)`, `Test and Lint (http3)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate on these events; skipped on PRs, merge groups and main pushes | `cargo nextest run` with the job's `--features` |
| Full selection, every event | `Test and Lint (gateway-stack)` | `ci.yml` `test-and-lint-gateway-stack` | Via aggregate | `cargo clippy -p rustfs --all-targets --features gateway-stack -- -D warnings`; `cargo nextest run -p rustfs --features gateway-stack` with the job's filter |
| Nightly schedule, manual dispatch, full selection | `Connect Feature Boundaries` | `ci.yml` `connect-feature-boundaries` | Via aggregate on these events; skipped on PRs, merge groups and main pushes | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `scripts/check_connect_release_boundary.py`: `cargo check -p rustfs --release` with `connect-e2e-short-credentials` and with `gateway-stack` must each fail at its guard; `scripts/check_offline_enrollment_e2e.sh` |
| Nightly schedule, manual dispatch, full selection | `Connect Feature Boundaries` | `ci.yml` `connect-feature-boundaries` | Via aggregate on these events; skipped on PRs, merge groups and main pushes | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `scripts/check_connect_release_boundary.py`: `cargo check -p rustfs --release` with `connect-e2e-short-credentials` must fail at its guard; `scripts/check_offline_enrollment_e2e.sh` |
| PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Via aggregate; prerequisite for black-box jobs | `python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks` (binary plus its `rustfs.e2e.json` sidecar) |
| Nightly schedule, manual dispatch, full selection | `io_uring Integration (real)` | `ci.yml` `uring-integration` | Via aggregate on these events; skipped on PRs, merge groups and main pushes | `cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture` |
| PR, non-doc change | `End-to-End Tests` | `ci.yml` `e2e-tests` | Via aggregate | `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test`, then `./scripts/e2e-run.sh ./target/debug/rustfs <data-dir>` under the same wrapper; membership guards `scripts/check_test_wiring.py --check-profile e2e-smoke <listing.json>` and `scripts/check_security_smoke_count.sh check <listing.json>` |
| PR, non-doc change | `Core contract` | `ci.yml` `core-contract` | Via aggregate | existing write/read, anonymous denial and restart-byte tests via `core-contract` profile and the E2E archive |
| Push to `main` or `gateway/integration`, merge group, nightly schedule, manual dispatch; PR only when it changes a gateway-stack input (`GATEWAY_SMOKE_PATHS` or a `rustfs-gateway*` pin, `scripts/ci_gate.py`) | `e2e-smoke (gateway, report-only until T2.10)` | `ci.yml` `e2e-tests-gateway` | Report-only until rustfs/backlog#2734 T2.10 makes it required; `continue-on-error`, outside the aggregate; skipped, never green, on PRs that touch no gateway-stack input. At T2.10, when the lane becomes required, it must run on every pull request again — delete this selection, do not extend it | `RUSTFS_S3_STACK=gateway python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke -p e2e_test`; `scripts/ci_gate.py` keeps its archive and run steps identical to `e2e-tests`, which sets `RUSTFS_S3_STACK=legacy` on this branch |
| PR, non-doc change | `S3 Implemented Tests` | `ci.yml` `s3-implemented-tests` | Via aggregate | build `rustfs`, then `scripts/s3-tests/run.sh` with the job's `DEPLOY_MODE` / `TEST_MODE` / `MAXFAIL` env |
| PR, non-doc change | `S3 Lifecycle Behavior Tests` | `ci.yml` `s3-lifecycle-behavior-tests` | Via aggregate | `scripts/s3-tests/run.sh` with the job's accelerated-scanner env |
| PR touching `paths` in `audit.yml` | `Cargo Deny`, `Workflow Pin Report`, `Dependency Review` | `audit.yml` `cargo-deny`, `workflow-pin-report`, `dependency-review` | Report-only | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
@@ -93,7 +91,7 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run. The v
| `cache-warm.yml` (every six hours, plus main pushes that change `Cargo.lock`, `rust-toolchain.toml` or the setup action) | `warm-ci-dev`, `warm-ci-feat-rio`, `warm-ci-feat-proto`, `warm-ci-uring` | no verdict; sole writer of the `rust-cache` keys that `ci.yml` restores | yes | dispatch `cache-warm.yml` |
| `build.yml` (nightly Linux, weekly desktop) | `build-rustfs` over the `prepare-platform-matrix` subset for the firing cron: four Linux targets at 01:13 UTC, and macOS aarch64 plus Windows x86_64 at Sunday 16:17 UTC | build/package integrity | yes | dispatch `build.yml` with an exact platform set |
| `e2e-replication-nightly.yml` (nightly) | `repl-nightly`, `cluster-nightly`, `protocols-nightly` | three independent gates; JUnit, membership listing, server logs | yes | `python3 scripts/e2e_binary.py build --bins`, then `python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-repl-nightly -p e2e_test`; `--features e2e-test-hooks` on both steps with `--profile e2e-nightly`; `--features ftps,webdav,sftp` on both steps with `-j 1 --profile e2e-protocols` |
| `dual-stack-nightly.yml` (nightly) | `build`, `leg` (`e2e-smoke` and `e2e-full`, each on the legacy and, once a gateway start is proven, the gateway stack), `summary` | report-only both-stack diff of the two profiles only `ci.yml` runs; the other e2e profiles carry the same `stack` dimension in their own workflows. A gateway binary that refuses to start skips the gateway legs and the summary says why; the gateway legs are report-only until T2.10 | yes | `python3 scripts/dual_stack.py summary --help`; dispatch `dual-stack-nightly.yml` |
| `dual-stack-nightly.yml` (nightly) | `build`, `leg` (`e2e-smoke` and `e2e-full`) | the two profiles only `ci.yml` runs, nightly on the gateway stack (the workflow keeps its name from the both-stack period); per-profile JUnit and server logs retained 30 days. The one comparison against the last legacy record is made by hand with `scripts/junit_diff.py` | yes | dispatch `dual-stack-nightly.yml` |
| `e2e-distributed.yml` (nightly) | `distributed` | fail-closed 4-node 4-disk S3, durability, replication, movement, fault, and direct/rolling upgrade gate; JUnit, membership listing, per-node server logs | yes | download the pinned previous release as in the workflow, export `RUSTFS_UPGRADE_SOURCE_BINARY`, run `python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks`, then `python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-distributed -p e2e_test` |
| `e2e-s3tests.yml` (weekly) | `s3tests` (single and distributed, four shards each), `upstream-head-canary` | compatibility gate; report, JUnit, node IDs, server logs | yes | `scripts/s3-tests/run.sh` against an existing single or distributed target |
| `fuzz.yml` (nightly) | `nightly-fuzz-corpus` per target | gate; corpus and crash artifacts | yes (disabled) | `MAX_TOTAL_TIME=<seconds> ./scripts/fuzz/run.sh` |
@@ -101,12 +99,12 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run. The v
| `on-demand-migration-interop.yml` (nightly) | `minio-source`, `cloud-source` (`aws`, `r2`, `gcs`) | report-only provider interop; one JSON report per provider naming cases, timings and source request counts, plus JUnit and MinIO logs. A cloud provider whose `ODM_INTEROP_*` secrets are absent is skipped with a summary note, not failed | yes | start the pinned MinIO container as in the job, export the `RUSTFS_ODM_INTEROP_*` variables, run `python3 scripts/e2e_binary.py build --bins`, then `python3 scripts/e2e_binary.py run -- cargo nextest run --profile e2e-odm-interop -p e2e_test` |
| `nightly-gnu.yml` (nightly) | `build`, `kms-vault-lane`, `kms-vault-ha-failover` | build, live Vault, and HA failover gates | yes | commands and pinned Vault images in the workflow |
| `audit.yml` (nightly) | `cargo-deny`, `workflow-pin-report` | dependency and workflow-pin gates | yes | `cargo deny check`; `scripts/security/check_workflow_pins.sh` |
| `mint.yml` (weekly) | `image`, `mint` (legacy and gateway), `compare` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json` per stack. The image is built once. `compare` holds the gate that every case legacy passes is present and passing on gateway and prints the cases that break it; the gateway leg and `compare` are report-only until T2.10 | yes | pinned Docker sequence in the workflow; `python3 scripts/mint_stack_diff.py <legacy-leg-dir> <gateway-leg-dir>` |
| `mint.yml` (weekly) | `image`, `mint` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json`. The image is built once | yes | pinned Docker sequence in the workflow |
| `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes (disabled) | `make coverage` |
| `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch |
| `e2e-upgrade.yml` (weekly, and on version tags) | `upgrade` (serial cases sharing one harness build) | upgrade and rollback gate; server logs | yes | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` commands with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) |
| `reference-parity.yml` (nightly, and on PRs that change the lane's own files) | `plan`, `build-candidate`, `s3tests` (one job per target in `.config/reference-parity-targets.toml`), `compare` | report-only parity gate: fails on any per-node-id s3-tests difference between a branch candidate and the release pinned in `.config/reference-binary.toml` that `.config/reference-parity-tolerated.txt` does not list, and on a stale tolerated entry; per-target JUnit, logs and `diff.md`/`diff.json` | yes | `DEPLOY_MODE=binary RUSTFS_BINARY=<binary> MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh` once per target, then `python3 scripts/s3-tests/reference_parity.py compare` as in the `compare` job |
| `gateway-acceptance.yml` (nightly) | `plan`, `build-candidate`, `build-gateway-tools`, `acceptance` (one job per S3 stack), `compare` | the rustfs/gateway conformance corpus (`--profile rustfs`) and client matrix (`--external`), both checked out at the rev `Cargo.toml` pins. Each stack must be observed in the server's own `server_starting` log event (`scripts/dual_stack.py stack`). Legacy stack: the corpus run is a reference that is never judged (it must serve, stay up and complete); the matrix is the baseline and fails on a `fail` or `sut-unregistered` cell or a client that could not be installed. Gateway stack, report-only until T2.10: fails on a conformance failure without a `.config/conformance-rulings.toml` row, on a ruled case failing differently, on a stale row, on a skip on the server's own answer, and on any matrix cell that passes on legacy but not on gateway. Per-stack `report.json`, `matrix.json` and logs, and a legacy-vs-gateway diff | yes | start the candidate with `RUSTFS_S3_STACK=<stack>`, then the job's `rustfs-gateway-conformance run` and `ci/compat/run_matrix.sh --external` commands, then `python3 scripts/check_conformance_rulings.py` and `python3 scripts/compat_matrix_report.py check` |
| `gateway-acceptance.yml` (nightly) | `plan`, `build-candidate`, `build-gateway-tools`, `acceptance` | the rustfs/gateway conformance corpus (`--profile rustfs`) and client matrix (`--external`), both checked out at the rev `Cargo.toml` pins, against one candidate server. Fails on a conformance failure without a `.config/conformance-rulings.toml` row, on a ruled case failing differently, on a stale row, on a skip on the server's own answer, and on a matrix `fail` or `sut-unregistered` cell or a client that could not be installed. `report.json`, `matrix.json` and logs | yes | start the candidate, then the job's `rustfs-gateway-conformance run` and `ci/compat/run_matrix.sh --external` commands, then `python3 scripts/check_conformance_rulings.py --run-dir <dir>` and `python3 scripts/compat_matrix_report.py check --run-dir <dir>` |
| `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | yes | see the PR row |
| `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | yes (disabled) | see the PR row |
| `gateway-integration-sync.yml` (weekdays) | `sync` | merges `main` into `gateway/integration` as a `--no-ff` merge commit; a conflicting merge is aborted and opens a `sync/<source>-<date>` pull request listing the conflicting files, which keeps the run green; a rejected push or missing pull request fails it | yes | dispatch, optionally with `source_ref` |
+6 -15
View File
@@ -56,12 +56,10 @@ required-features = ["swift"]
name = "swift_object_integration_test"
required-features = ["swift"]
# Starts an embedded server on the gateway stack, which only a build with the
# feature can select.
# Starts an embedded server on the gateway stack.
[[test]]
name = "embedded_gateway_stack_test"
path = "tests/embedded_gateway_stack_test/main.rs"
required-features = ["gateway-stack"]
[features]
internode-transport = ["rustfs-ecstore/internode-transport", "rustfs-rio/internode-transport"]
@@ -77,19 +75,12 @@ license = []
io-scheduler-debug = [] # Enable debug information in I/O scheduler
tracing-chunk-debug = [] # Enable per-chunk tracing in data plane (high noise, for debugging only)
full = ["metrics-gpu", "ftps", "swift", "webdav", "sftp", "pyroscope", "gcs"]
# The debug-only E2E binary also carries the gateway S3 stack, so the gateway smoke
# lane runs the same artifact as the legacy one with RUSTFS_S3_STACK as its only change.
e2e-test-hooks = ["rustfs-ecstore/e2e-test-hooks", "rustfs-scanner/e2e-test-hooks", "gateway-stack"]
e2e-test-hooks = ["rustfs-ecstore/e2e-test-hooks", "rustfs-scanner/e2e-test-hooks"]
# Shortens Connect credentials only in debug E2E builds.
connect-e2e-short-credentials = []
# Builds the dedicated rustfs-cli-e2e target with a build-time public enrollment root.
offline-enrollment-e2e-root = []
rio-v2 = ["rustfs-ecstore/rio-v2"]
# The rustfs-gateway S3 stack, selected at startup with RUSTFS_S3_STACK=gateway
# (rustfs/backlog#2734). Under construction: off by default and never enabled in a
# release build; the stack refuses to compile without debug assertions, and
# `scripts/check_connect_release_boundary.py` keeps every release-reachable feature from enabling it.
gateway-stack = ["dep:rustfs-gateway", "dep:rustfs-gateway-adapter", "dep:rustfs-gateway-core", "dep:rustfs-gateway-dialect-rustfs-admin"]
# Linux-only cuObject S3-over-RDMA GET into client (GPU) memory; needs libcuobjserver.
cuobject = ["dep:rustfs-cuobject", "rustfs-cuobject/server"]
# NIXL/UCX transport for S3-over-RDMA; replaces cuobject when both are enabled.
@@ -279,10 +270,10 @@ rustfs-notify = { workspace = true }
rustfs-obs = { workspace = true }
rustfs-policy = { workspace = true }
rustfs-protocols = { workspace = true }
rustfs-gateway = { workspace = true, optional = true }
rustfs-gateway-adapter = { workspace = true, optional = true }
rustfs-gateway-core = { workspace = true, optional = true }
rustfs-gateway-dialect-rustfs-admin = { workspace = true, optional = true }
rustfs-gateway = { workspace = true }
rustfs-gateway-adapter = { workspace = true }
rustfs-gateway-core = { workspace = true }
rustfs-gateway-dialect-rustfs-admin = { workspace = true }
rustfs-gateway-stream = { workspace = true }
rustfs-gateway-types = { workspace = true, features = ["compat-s3s-rustfs"] }
rustfs-protos = { workspace = true }
+3 -2
View File
@@ -20,7 +20,7 @@ use crate::admin::service::caller_identity::oidc_profile_fields;
use crate::admin::storage_api::bucket::versioning_sys::BucketVersioningSys;
use crate::admin::storage_api::contract::admin::StorageAdminApi;
use crate::admin::storage_api::contract::bucket::{BucketOperations, BucketOptions};
use crate::admin::storage_api::data_usage::load_admin_data_usage_with_memory_overlay;
use crate::admin::storage_api::data_usage::{apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached};
use crate::admin::storage_api::metadata_sys;
use crate::auth::get_condition_values;
use crate::server::ADMIN_PREFIX;
@@ -278,7 +278,8 @@ impl Operation for AccountInfoHandler {
// process-local absolute counters to cluster-wide bucket totals.
// This path never triggers a live full-version listing
// (rustfs/backlog#1306); freshness is owned by the scanner.
let data_usage_info = map_data_usage_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
let mut data_usage_info = map_data_usage_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
apply_bucket_usage_memory_overlay(&mut data_usage_info).await;
account_info.usage_snapshot_converged = data_usage_info.usage_snapshot_converged;
for bucket in buckets.iter() {
@@ -1261,7 +1261,16 @@ async fn table_storage_credentials_are_rejected_by_catalog_authentication() {
}
let mut credential = get_new_credentials_with_metadata(&claims, root_secret).expect("sign a temporary credential");
credential.parent_user = parent.to_string();
iam.set_temp_user(&credential.access_key, &credential, None)
// A regular parent's sessions must carry its durable STS generation, so publish
// through the parent-bound path instead of storing an unversioned session.
let parent_secret = if parent == root_key { root_secret } else { parent_secret };
let authenticated_parent = rustfs_credentials::Credentials {
access_key: parent.to_string(),
secret_key: parent_secret.to_string(),
status: "on".to_string(),
..Default::default()
};
iam.set_temp_user_from_parent(&credential, &authenticated_parent)
.await
.expect("store temporary credential");
let request = load_table_request_builder(context.clone(), &credential.access_key, &credential.secret_key)
-5
View File
@@ -916,14 +916,12 @@ async fn handle_object_lambda_request(req: &mut S3Request<Body>, bucket: &str, o
invoke_object_lambda_target(req, bucket, object, get_resp).await
}
#[cfg(feature = "gateway-stack")]
/// The bucket and key of an Object Lambda request, which takes precedence over every other route
/// this router claims; `None` for any other request.
pub(crate) fn object_lambda_target(method: &Method, uri: &Uri) -> Option<(String, String)> {
parse_object_lambda_request(method, uri)
}
#[cfg(feature = "gateway-stack")]
/// Serves an admitted Object Lambda request that another edge verified, through the same handler
/// the legacy `call` runs.
pub(crate) async fn serve_object_lambda(
@@ -973,7 +971,6 @@ pub(crate) fn console_listener_route(method: &Method, uri: &Uri, headers: &Heade
/// Whether this router answers the request itself rather than an S3 operation: the RustFS extension
/// routes and every control-plane route.
#[cfg(feature = "gateway-stack")]
pub(crate) fn claims_route(method: &Method, uri: &Uri, headers: &HeaderMap) -> bool {
s3_extension::route_key(method, uri).is_some()
|| parse_object_lambda_request(method, uri).is_some()
@@ -981,7 +978,6 @@ pub(crate) fn claims_route(method: &Method, uri: &Uri, headers: &HeaderMap) -> b
}
/// Signed system-status and catalog-configuration reads, after object-lambda routing.
#[cfg(feature = "gateway-stack")]
pub(crate) fn claims_native_admin_route(method: &Method, uri: &Uri) -> bool {
method == Method::GET
&& matches!(
@@ -1011,7 +1007,6 @@ pub(crate) fn claims_native_admin_route(method: &Method, uri: &Uri) -> bool {
}
/// Console paths whose requests do not belong to an earlier extension route.
#[cfg(feature = "gateway-stack")]
pub(crate) fn claims_native_console_route(method: &Method, uri: &Uri) -> bool {
is_console_path(uri.path())
&& parse_object_lambda_request(method, uri).is_none()
+4 -6
View File
@@ -945,6 +945,10 @@ pub(crate) static ERR_TIER_RESERVED_NAME: AdminErrorRef = AdminErrorRef(|| &ecst
pub(crate) mod data_usage {
use std::sync::Arc;
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_data_usage_from_backend(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
@@ -956,12 +960,6 @@ pub(crate) mod data_usage {
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
}
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
}
}
pub(crate) mod access {
+22 -65
View File
@@ -21,7 +21,9 @@ use super::storage_api::admin_usecase::capacity::{
};
use super::storage_api::admin_usecase::contract::StorageAdminApi;
use super::storage_api::admin_usecase::contract::bucket::{BucketOperations as _, BucketOptions};
use super::storage_api::admin_usecase::data_usage::load_admin_data_usage_with_memory_overlay;
use super::storage_api::admin_usecase::data_usage::{
apply_bucket_usage_memory_overlay, load_admin_data_usage_from_backend_cached,
};
use super::storage_api::admin_usecase::{ECStore, EndpointServerPools};
use crate::app::runtime_sources::{
AppContext, current_app_context, current_endpoints_handle, current_object_store_handle_for_context,
@@ -270,7 +272,7 @@ impl DefaultAdminUsecase {
info.buckets_usage.retain(|bucket, _| current_buckets.contains(bucket));
info.bucket_sizes.retain(|bucket, _| current_buckets.contains(bucket));
if info.buckets_usage.is_empty() && !current_buckets.is_empty() && !info.usage_snapshot_admin_projection {
if info.buckets_usage.is_empty() && !current_buckets.is_empty() {
*info = DataUsageInfo::default();
return;
}
@@ -285,14 +287,19 @@ impl DefaultAdminUsecase {
}
}
/// Keep already identity-bound display values without accepting a display
/// projection as a persistable or quota-authoritative snapshot.
fn present_identity_bound_data_usage_snapshot(
mut info: DataUsageInfo,
buckets: impl IntoIterator<Item = String>,
) -> DataUsageInfo {
let snapshot_measured =
info.usage_snapshot_admin_projection || info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
/// Overlay single-process counts on a persisted snapshot and narrow it
/// for display.
///
/// Whether the snapshot measured anything is decided before the overlay.
/// The overlay clears `usage_snapshot_complete` when it holds counts it
/// cannot reconcile with the snapshot, such as writes into a bucket that
/// has no verified baseline. That makes the totals unconverged, not the
/// scanned buckets unusable; treating it as structural incompleteness
/// blanked usage for every bucket for as long as one such bucket kept
/// receiving writes, because each write restarts its scanner fence.
async fn present_data_usage_snapshot(mut info: DataUsageInfo, buckets: impl IntoIterator<Item = String>) -> DataUsageInfo {
let snapshot_measured = info.is_complete_bucket_usage_snapshot() || info.is_valid_partial_snapshot();
apply_bucket_usage_memory_overlay(&mut info).await;
Self::narrow_data_usage_snapshot_to_measured_buckets(&mut info, snapshot_measured, buckets);
info
}
@@ -338,7 +345,7 @@ impl DefaultAdminUsecase {
/// This path never triggers a live full-version listing
/// (rustfs/backlog#1306); freshness is owned by the scanner.
pub(crate) async fn query_data_usage_info_with_store(store: Arc<ECStore>) -> AdminUsecaseResult<DataUsageInfo> {
let info = Self::map_data_usage_load_result(load_admin_data_usage_with_memory_overlay(store.clone()).await)?;
let info = Self::map_data_usage_load_result(load_admin_data_usage_from_backend_cached(store.clone()).await)?;
let buckets = store
.list_bucket(&BucketOptions {
cached: true,
@@ -347,7 +354,7 @@ impl DefaultAdminUsecase {
})
.await
.map_err(|_| Self::app_error(S3ErrorCode::InternalError, "list_bucket failed"))?;
let mut info = Self::present_identity_bound_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name));
let mut info = Self::present_data_usage_snapshot(info, buckets.into_iter().map(|bucket| bucket.name)).await;
let storage_info = StorageAdminApi::storage_info(store.as_ref()).await;
@@ -972,52 +979,6 @@ mod tests {
assert_eq!(info.buckets_count, 0);
}
#[test]
fn identity_bound_partial_display_keeps_valid_buckets_and_unknown_state() {
use std::collections::HashMap;
let mut projected = DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
usage_snapshot_admin_projection: true,
usage_snapshot_partial: true,
usage_snapshot_converged: Some(false),
buckets_count: 1,
objects_total_count: 7,
objects_total_size: 70,
buckets_usage: HashMap::from([(
"cold".to_string(),
rustfs_data_usage::BucketUsageInfo {
size: 70,
objects_count: 7,
..Default::default()
},
)]),
bucket_sizes: HashMap::from([("cold".to_string(), 70)]),
..Default::default()
};
projected = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
projected,
["cold".to_string(), "recreated".to_string()],
);
assert_eq!(projected.buckets_usage["cold"].objects_count, 7);
assert!(!projected.buckets_usage.contains_key("recreated"));
assert_eq!(projected.objects_total_count, 7);
assert_eq!(projected.usage_snapshot_converged, Some(false));
assert!(!projected.is_complete_bucket_usage_snapshot());
assert!(!projected.is_valid_partial_snapshot());
projected.buckets_usage.clear();
projected.bucket_sizes.clear();
projected.calculate_totals();
projected.buckets_count = 0;
let unknown = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(projected, ["recreated".to_string()]);
assert!(unknown.buckets_usage.is_empty());
assert!(unknown.usage_snapshot_admin_projection);
assert_eq!(unknown.usage_snapshot_converged, Some(false));
assert!(!unknown.is_complete_bucket_usage_snapshot());
assert_eq!(unknown.last_update, Some(std::time::SystemTime::UNIX_EPOCH));
}
/// A write into a bucket without a verified baseline leaves process-local
/// counts the overlay cannot reconcile, so it clears
/// `usage_snapshot_complete`. That must flag the response non-converged,
@@ -1048,15 +1009,11 @@ mod tests {
crate::app::storage_api::test::data_usage::record_bucket_object_write_memory(unverified_bucket, None, 1_024).await;
// Exercise the production overlay, followed by the same identity-bound
// presentation function used by the admin query.
crate::app::storage_api::test::data_usage::apply_bucket_usage_memory_overlay(&mut persisted).await;
persisted.usage_snapshot_admin_projection = true;
persisted.usage_snapshot_converged = Some(false);
let info = DefaultAdminUsecase::present_identity_bound_data_usage_snapshot(
let info = DefaultAdminUsecase::present_data_usage_snapshot(
persisted,
[measured_bucket.to_string(), unverified_bucket.to_string()],
);
)
.await;
assert_eq!(
info.buckets_usage
+1 -2
View File
@@ -22,7 +22,7 @@ use super::storage_api::bucket_usecase::StorageObjectInfo as ObjectInfo;
#[cfg(test)]
use super::storage_api::bucket_usecase::access::ReqInfo;
use super::storage_api::bucket_usecase::access::{
TableDataPlaneListAccess, authorize_request, bucket_config_mutation_incarnation, log_list_buckets_iam_implicit_deny,
authorize_request, bucket_config_mutation_incarnation, log_list_buckets_iam_implicit_deny,
prepare_list_buckets_iam_authorization, prepare_odm_read_generation, req_info_ref,
};
#[cfg(test)]
@@ -121,7 +121,6 @@ use tracing::{Instrument as _, debug, error, info, instrument, warn};
mod accelerate;
mod acl;
mod core;
#[cfg(any(test, feature = "gateway-stack"))]
pub(crate) use self::core::requested_location_constraint;
mod cors;
#[cfg(test)]
+52 -37
View File
@@ -22,7 +22,9 @@ use crate::app::storage_api::s3::dto::{
UserMetadata,
};
use crate::app::storage_api::s3::xml::{SerResult, SerializeContent};
use crate::app::storage_api::s3::{S3Request, S3Response, S3Result, xml};
#[cfg(test)]
use crate::app::storage_api::s3::{S3ErrorCode, S3Request, S3Response};
use crate::app::storage_api::s3::{S3Result, xml};
type ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>;
@@ -320,31 +322,20 @@ fn metadata_count_to_i32(value: usize) -> i32 {
i32::try_from(value).unwrap_or(i32::MAX)
}
async fn is_list_objects_metadata_action_allowed<T>(
req: &S3Request<T>,
async fn is_list_objects_metadata_action_allowed(
envelope: &RequestEnvelope,
bucket: &str,
object: &str,
action: S3Action,
) -> S3Result<bool> {
let mut auth_req = S3Request {
input: (),
method: req.method.clone(),
uri: req.uri.clone(),
headers: req.headers.clone(),
extensions: req.extensions.clone(),
credentials: req.credentials.clone(),
region: req.region.clone(),
service: req.service.clone(),
trailing_headers: req.trailing_headers.clone(),
};
let mut req_info = req_info_ref(req)?.clone();
let mut auth_req = envelope.fork_for_authorization();
let mut req_info = req_info_ref(envelope)?.clone();
req_info.bucket = Some(bucket.to_string());
req_info.object = Some(object.to_string());
req_info.version_id = None;
// Denial here is an expected filter outcome, not an error (issue #5740).
req_info.suppress_denial_log = true;
auth_req.extensions.insert(req_info);
auth_req.install_req_info(req_info);
match authorize_request(&mut auth_req, Action::S3Action(action)).await {
Ok(()) => Ok(true),
@@ -353,8 +344,8 @@ async fn is_list_objects_metadata_action_allowed<T>(
}
}
async fn collect_list_objects_metadata_permissions<T>(
req: &S3Request<T>,
async fn collect_list_objects_metadata_permissions(
envelope: &RequestEnvelope,
bucket: &str,
objects: &[ObjectInfo],
) -> S3Result<HashMap<String, ObjectMetadataPermissions>> {
@@ -366,9 +357,9 @@ async fn collect_list_objects_metadata_permissions<T>(
}
let metadata_allowed =
is_list_objects_metadata_action_allowed(req, bucket, &object.name, S3Action::GetObjectAction).await?;
is_list_objects_metadata_action_allowed(envelope, bucket, &object.name, S3Action::GetObjectAction).await?;
let tags_allowed =
is_list_objects_metadata_action_allowed(req, bucket, &object.name, S3Action::GetObjectTaggingAction).await?;
is_list_objects_metadata_action_allowed(envelope, bucket, &object.name, S3Action::GetObjectTaggingAction).await?;
permissions.insert(
object.name.clone(),
@@ -605,11 +596,22 @@ fn build_list_objects_v2_metadata_output(
}
impl DefaultBucketUsecase {
#[cfg(test)]
pub(crate) async fn execute_list_objects_v2m(
&self,
req: S3Request<ListObjectsV2Input>,
) -> S3Result<S3Response<ListObjectsV2MetadataOutput>> {
let input = req.input.clone();
let envelope = RequestEnvelope::from_s3s(&req);
self.list_objects_v2_metadata(&envelope, req.input).await.map(S3Response::new)
}
/// One page of `GET /{bucket}?list-type=2&metadata=true`, for the caller the envelope's
/// authorization installed: each entry's metadata and tags as far as that caller may read them.
pub(crate) async fn list_objects_v2_metadata(
&self,
envelope: &RequestEnvelope,
input: ListObjectsV2Input,
) -> S3Result<ListObjectsV2MetadataOutput> {
let ListObjectsV2Input {
bucket,
continuation_token,
@@ -624,14 +626,14 @@ impl DefaultBucketUsecase {
let params = parse_list_objects_v2_params(prefix, delimiter, max_keys, continuation_token, start_after)?;
validate_list_object_unordered_with_delimiter(params.delimiter.as_ref(), req.uri.query())?;
validate_list_object_unordered_with_delimiter(params.delimiter.as_ref(), envelope.uri().query())?;
let store = get_validated_store(&bucket).await?;
let incl_deleted = get_header(&req.headers, rustfs_utils::http::SUFFIX_INCLUDE_DELETED)
let incl_deleted = get_header(envelope.headers(), rustfs_utils::http::SUFFIX_INCLUDE_DELETED)
.map(|value| value.as_ref() == "true")
.unwrap_or_default();
let object_infos = match req.extensions.get::<TableDataPlaneListAccess>() {
let object_infos = match envelope.table_list_access() {
Some(access) => {
table_list_isolation::list_objects_v2(
store.clone(),
@@ -664,24 +666,34 @@ impl DefaultBucketUsecase {
.map_err(ApiError::from)?,
};
let permissions = collect_list_objects_metadata_permissions(&req, &bucket, &object_infos.objects).await?;
let output = build_list_objects_v2_metadata_output(
let permissions = collect_list_objects_metadata_permissions(envelope, &bucket, &object_infos.objects).await?;
Ok(build_list_objects_v2_metadata_output(
object_infos,
&bucket,
&params,
encoding_type.as_ref(),
fetch_owner.unwrap_or_default(),
&permissions,
);
Ok(S3Response::new(output))
))
}
#[cfg(test)]
pub(crate) async fn execute_list_object_versions_m(
&self,
req: S3Request<ListObjectVersionsInput>,
) -> S3Result<S3Response<ListObjectVersionsMetadataOutput>> {
let input = req.input.clone();
let envelope = RequestEnvelope::from_s3s(&req);
self.list_object_versions_metadata(&envelope, req.input)
.await
.map(S3Response::new)
}
/// One page of `GET /{bucket}?versions&metadata=true`, as [`Self::list_objects_v2_metadata`].
pub(crate) async fn list_object_versions_metadata(
&self,
envelope: &RequestEnvelope,
input: ListObjectVersionsInput,
) -> S3Result<ListObjectVersionsMetadataOutput> {
let ListObjectVersionsInput {
bucket,
delimiter,
@@ -696,7 +708,7 @@ impl DefaultBucketUsecase {
let params = parse_list_object_versions_params(prefix, delimiter, key_marker, version_id_marker, max_keys)?;
let store = get_validated_store(&bucket).await?;
let object_infos = match req.extensions.get::<TableDataPlaneListAccess>() {
let object_infos = match envelope.table_list_access() {
Some(access) => {
table_list_isolation::list_object_versions(
store.clone(),
@@ -725,11 +737,14 @@ impl DefaultBucketUsecase {
.map_err(ApiError::from)?,
};
let permissions = collect_list_objects_metadata_permissions(&req, &bucket, &object_infos.objects).await?;
let output =
build_list_object_versions_metadata_output(object_infos, &bucket, &params, encoding_type.as_ref(), &permissions);
Ok(S3Response::new(output))
let permissions = collect_list_objects_metadata_permissions(envelope, &bucket, &object_infos.objects).await?;
Ok(build_list_object_versions_metadata_output(
object_infos,
&bucket,
&params,
encoding_type.as_ref(),
&permissions,
))
}
}
@@ -457,3 +457,130 @@ fn legacy_put_notification_does_not_write_after_the_bucket_is_recreated() {
);
});
}
#[test]
fn notification_dispatch_follows_peer_reload_delete_and_recreation() {
run_large_stack_test("notification-metadata-authority", || async {
let ambient = shared_gating_ambient().await;
let store = ambient.object_store();
let bucket = format!("notify-authority-{}", uuid::Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let xml = |event: &str| {
format!(
"<NotificationConfiguration><QueueConfiguration><Event>{event}</Event>\
<Queue>arn:rustfs:sqs:us-east-1:inert:webhook</Queue></QueueConfiguration></NotificationConfiguration>"
)
.into_bytes()
};
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, xml("s3:ObjectCreated:Put"))
.await
.expect("persist initial subscription");
let old = store.get_bucket_metadata(&bucket).await.expect("initial cached metadata");
let system = rustfs_notify::NotificationSystem::new(Default::default());
system.bind_bucket_metadata_source(&store);
let engine = rustfs_notify::NotifyRuleEngine::new();
engine.bind_bucket_metadata_source(&store);
let inert_target = TargetID::new("inert".into(), "webhook".into());
assert!(system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
assert!(engine.is_target_bound_to_any_bucket(&inert_target).await);
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, EMPTY_XML.to_vec())
.await
.expect("remove subscription durably");
// Stand in for a peer whose cache has not received the reload yet.
crate::storage::storage_api::set_bucket_metadata(bucket.clone(), (*old).clone())
.await
.expect("install stale peer cache");
assert!(system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
crate::storage::storage_api::reload_bucket_metadata(store.clone(), &bucket)
.await
.expect("peer reload");
assert!(!system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
assert!(
engine
.match_targets(&bucket, EventName::ObjectCreatedPut, "object")
.await
.is_empty()
);
assert!(!engine.is_target_bound_to_any_bucket(&inert_target).await);
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, xml("s3:ObjectCreated:Put"))
.await
.expect("restore initial subscription");
assert!(system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
store
.delete_bucket(&bucket, &DeleteBucketOptions::default())
.await
.expect("delete bucket");
assert!(!system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
// A peer may still have the deleted incarnation cached until its delete
// callback (or the periodic metadata refresh) establishes absence.
crate::storage::storage_api::set_bucket_metadata(bucket.clone(), (*old).clone())
.await
.expect("install deleted peer snapshot");
assert!(system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
assert!(
crate::storage::storage_api::remove_bucket_metadata_if_absent(store.clone(), &bucket)
.await
.expect("peer deletion")
);
assert!(!system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
assert!(
engine
.match_targets(&bucket, EventName::ObjectCreatedPut, "object")
.await
.is_empty()
);
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
assert_ne!(
old.bucket_incarnation_id,
store.bucket_incarnation_id(&bucket).await.expect("replacement identity")
);
assert!(!system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
store
.update_bucket_metadata_config(&bucket, BUCKET_NOTIFICATION_CONFIG, xml("s3:ObjectRemoved:Delete"))
.await
.expect("replacement subscription");
assert!(system.has_subscriber(&bucket, &EventName::ObjectRemovedDelete).await);
assert!(engine.has_subscriber(&bucket, &EventName::ObjectRemovedDelete).await);
assert!(
!crate::storage::storage_api::remove_bucket_metadata_if_absent(store.clone(), &bucket)
.await
.expect("delayed peer deletion")
);
let mut stale = rustfs_notify::BucketNotificationConfig::new("us-east-1");
stale.add_rule(&[EventName::ObjectCreatedPut], "*".into(), TargetID::new("old".into(), "webhook".into()));
// No runtime targets are activated: all assertions inspect inert rules.
system.remove_bucket_notification_config(&bucket).await;
// Applying already-validated rules models a delayed PUT completion.
engine.set_bucket_rules(&bucket, stale.get_rules_map().clone()).await;
assert!(engine.has_subscriber(&bucket, &EventName::ObjectRemovedDelete).await);
assert!(!engine.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
assert!(
engine
.match_targets(&bucket, EventName::ObjectCreatedPut, "object")
.await
.is_empty()
);
assert_eq!(
engine
.match_targets(&bucket, EventName::ObjectRemovedDelete, "object")
.await
.into_iter()
.collect::<Vec<_>>(),
vec![inert_target]
);
assert!(system.has_subscriber(&bucket, &EventName::ObjectRemovedDelete).await);
assert!(!system.has_subscriber(&bucket, &EventName::ObjectCreatedPut).await);
});
}
-9
View File
@@ -198,7 +198,6 @@ pub struct RequestEnvelope {
website_read: Option<WebsiteRead>,
post_object: Option<PostObjectRequestMarker>,
gateway_post: bool,
#[cfg(feature = "gateway-stack")]
gateway_post_archive_identity: Option<Arc<()>>,
post_object_policy_headers: Option<PostObjectPolicyHeaders>,
// Attached by the authorization phase (`storage/access.rs`) for the use case.
@@ -252,7 +251,6 @@ impl RequestEnvelope {
website_read: extensions.remove(),
post_object: extensions.remove(),
gateway_post: false,
#[cfg(feature = "gateway-stack")]
gateway_post_archive_identity: None,
post_object_policy_headers: extensions.remove(),
req_info: extensions.remove(),
@@ -304,7 +302,6 @@ impl RequestEnvelope {
website_read: self.website_read.clone(),
post_object: self.post_object.clone(),
gateway_post: self.gateway_post,
#[cfg(feature = "gateway-stack")]
gateway_post_archive_identity: self.gateway_post_archive_identity.clone(),
post_object_policy_headers: self.post_object_policy_headers.clone(),
req_info: self.req_info.clone(),
@@ -465,7 +462,6 @@ impl RequestEnvelope {
&self.residual_extensions
}
#[cfg(feature = "gateway-stack")]
/// The payloads the connection, request and signature layers attached, back in the extensions
/// they arrive in, for an edge that hands this request to a handler still taking those.
pub(crate) fn transport_extensions(&self) -> Extensions {
@@ -528,13 +524,11 @@ impl RequestEnvelope {
self.gateway_post
}
#[cfg(feature = "gateway-stack")]
pub(crate) fn mark_gateway_post(&mut self) {
self.gateway_post = true;
self.gateway_post_archive_identity = None;
}
#[cfg(feature = "gateway-stack")]
pub(crate) fn bind_gateway_post_archive_identity(&mut self) -> Option<&Arc<()>> {
if !self.gateway_post {
return None;
@@ -542,7 +536,6 @@ impl RequestEnvelope {
Some(self.gateway_post_archive_identity.get_or_insert_with(|| Arc::new(())))
}
#[cfg(feature = "gateway-stack")]
pub(crate) fn gateway_post_identity(&self) -> Option<&Arc<()>> {
self.gateway_post_archive_identity.as_ref()
}
@@ -563,7 +556,6 @@ impl RequestEnvelope {
self.bucket_generation.as_ref()
}
#[cfg(feature = "gateway-stack")]
pub(crate) fn take_bucket_generation(&mut self) -> Option<BucketGenerationGuard> {
self.bucket_generation.take()
}
@@ -593,7 +585,6 @@ impl RequestEnvelope {
}
/// Releases preflight guards before uncommitted POST staging; publication must reauthorize.
#[cfg(feature = "gateway-stack")]
pub(crate) fn release_table_publication_guards(&mut self) {
self.table_publication_guards = None;
}
@@ -108,17 +108,6 @@ async fn persist_scanner_snapshot(ecstore: Arc<ECStore>, info: DataUsageInfo) {
rustfs_scanner::scanner::store_data_usage_in_backend(CancellationToken::new(), ecstore, receiver).await;
}
async fn bind_current_bucket_incarnations(ecstore: &ECStore, info: &mut DataUsageInfo) {
for (bucket, usage) in &mut info.buckets_usage {
let incarnation = ecstore
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("load the current incarnation for the covered snapshot");
assert!(!incarnation.is_nil(), "covered snapshot must bind a non-nil bucket identity");
usage.bucket_incarnation = Some(incarnation.to_string());
}
}
#[tokio::test]
#[serial]
async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
@@ -185,7 +174,6 @@ async fn data_usage_endpoint_serves_snapshot_without_live_listing() {
seeded_info.bucket_sizes.insert(overlay_bucket.clone(), 0);
seeded_info.buckets_count = u64::try_from(seeded_info.buckets_usage.len()).expect("test bucket count should fit into u64");
seeded_info.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut seeded_info).await;
persist_scanner_snapshot(ecstore.clone(), seeded_info).await;
record_bucket_object_write_memory(&overlay_bucket, None, 512).await;
@@ -353,7 +341,6 @@ async fn data_usage_endpoint_treats_legacy_partial_stats_as_unknown_until_covere
completed.bucket_sizes.insert(small_bucket.clone(), 57_344);
completed.buckets_count = u64::try_from(completed.buckets_usage.len()).expect("test bucket count should fit into u64");
completed.calculate_totals();
bind_current_bucket_incarnations(&ecstore, &mut completed).await;
store_data_usage_in_backend(completed, ecstore.clone())
.await
.expect("persist complete covered snapshot");
+267
View File
@@ -0,0 +1,267 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! RustFS's two metadata listings, `GET /{bucket}?list-type=2&metadata=true` and
//! `GET /{bucket}?versions&metadata=true`: their authorization and their answer.
//!
//! Responsible for: [`authorize`], the listing check both stacks run once the caller is
//! installed, and [`answer`], which reads the listing's parameters from the request, runs the
//! owned listing use case and renders the listing document with each entry's metadata, tags and
//! internal fields.
//! NOT responsible for: which requests are metadata listings (the legacy `MetadataRoute` and the
//! gateway's RustFS dialect each select them), resolving the caller, or the listing itself
//! (`bucket_usecase/listing.rs`).
//! Upstream: the legacy `MetadataRoute` and the gateway stack's metadata listing handler.
//! Downstream: the bucket listing authorization and the listing use case.
use crate::app::RequestEnvelope;
use crate::app::bucket_usecase::DefaultBucketUsecase;
use crate::app::storage_api::s3::dto::{EncodingType, ListObjectVersionsInput, ListObjectsV2Input};
use crate::app::storage_api::s3::xml;
use crate::storage::access::access_owned::listing_multipart::authorize_bucket_listing;
use crate::storage::authorization::AuthorizationRequest;
use http::{HeaderMap, Uri};
use rustfs_policy::policy::action::S3Action;
use rustfs_s3_types::{S3Error, S3Result, s3_error};
use url::form_urlencoded;
/// Which of the two metadata listings a request is.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub(crate) enum MetadataListing {
ListObjectVersions,
ListObjectsV2,
}
impl MetadataListing {
/// The action the listing is authorized by on its bucket.
fn action(self) -> S3Action {
match self {
Self::ListObjectVersions => S3Action::ListBucketVersionsAction,
Self::ListObjectsV2 => S3Action::ListBucketAction,
}
}
}
/// Authorizes `listing` of `bucket` for the caller `req` carries: the bucket listing check of
/// the standard listing with the same action, the table snapshot included.
///
/// # Errors
///
/// What the bucket listing authorization refuses.
pub(crate) async fn authorize<R: AuthorizationRequest + ?Sized>(
req: &mut R,
listing: MetadataListing,
bucket: &str,
) -> S3Result<()> {
authorize_bucket_listing(req, bucket, listing.action()).await
}
/// The listing document of `listing` over `bucket`, after [`authorize`] ran on `envelope`.
///
/// # Errors
///
/// `InvalidArgument` for a malformed listing parameter; otherwise what the listing use case
/// refuses.
pub(crate) async fn answer(envelope: &RequestEnvelope, listing: MetadataListing, bucket: String) -> S3Result<Vec<u8>> {
let usecase = DefaultBucketUsecase::from_global();
match listing {
MetadataListing::ListObjectVersions => {
let input = list_object_versions_input(bucket, envelope.uri(), envelope.headers())?;
render(&usecase.list_object_versions_metadata(envelope, input).await?)
}
MetadataListing::ListObjectsV2 => {
let input = list_objects_v2_input(bucket, envelope.uri(), envelope.headers())?;
render(&usecase.list_objects_v2_metadata(envelope, input).await?)
}
}
}
fn render<T: xml::Serialize>(output: &T) -> S3Result<Vec<u8>> {
let mut body = Vec::with_capacity(1024);
let mut serializer = xml::Serializer::new(&mut body);
serializer
.decl()
.and_then(|()| output.serialize(&mut serializer))
.map_err(S3Error::internal_error)?;
Ok(body)
}
fn list_object_versions_input(bucket: String, uri: &Uri, headers: &HeaderMap) -> S3Result<ListObjectVersionsInput> {
let query = uri.query().unwrap_or_default();
Ok(ListObjectVersionsInput {
bucket,
delimiter: query_value(query, "delimiter"),
encoding_type: parse_encoding_type(query_value(query, "encoding-type"))?,
expected_bucket_owner: header_value(headers, "x-amz-expected-bucket-owner")?,
key_marker: query_value(query, "key-marker"),
max_keys: query_i32(query, "max-keys")?,
optional_object_attributes: None,
prefix: query_value(query, "prefix"),
request_payer: header_value(headers, "x-amz-request-payer")?.map(Into::into),
version_id_marker: query_value(query, "version-id-marker"),
})
}
fn list_objects_v2_input(bucket: String, uri: &Uri, headers: &HeaderMap) -> S3Result<ListObjectsV2Input> {
let query = uri.query().unwrap_or_default();
Ok(ListObjectsV2Input {
bucket,
continuation_token: query_value(query, "continuation-token"),
delimiter: query_value(query, "delimiter"),
encoding_type: parse_encoding_type(query_value(query, "encoding-type"))?,
expected_bucket_owner: header_value(headers, "x-amz-expected-bucket-owner")?,
fetch_owner: query_bool(query, "fetch-owner")?,
max_keys: query_i32(query, "max-keys")?,
optional_object_attributes: None,
prefix: query_value(query, "prefix"),
request_payer: header_value(headers, "x-amz-request-payer")?.map(Into::into),
start_after: query_value(query, "start-after"),
})
}
/// The first value the query gives `key`.
pub(crate) fn query_value(query: &str, key: &str) -> Option<String> {
form_urlencoded::parse(query.as_bytes())
.find(|(name, _)| name == key)
.map(|(_, value)| value.into_owned())
}
fn query_i32(query: &str, key: &str) -> S3Result<Option<i32>> {
query_value(query, key)
.map(|value| {
value
.parse::<i32>()
.map_err(|_| s3_error!(InvalidArgument, "invalid integer query value"))
})
.transpose()
}
fn query_bool(query: &str, key: &str) -> S3Result<Option<bool>> {
query_value(query, key)
.map(|value| {
value
.parse::<bool>()
.map_err(|_| s3_error!(InvalidArgument, "invalid boolean query value"))
})
.transpose()
}
fn parse_encoding_type(value: Option<String>) -> S3Result<Option<EncodingType>> {
value
.map(|value| {
if value == EncodingType::URL {
Ok(EncodingType::from_static(EncodingType::URL))
} else {
Err(s3_error!(InvalidArgument, "invalid encoding-type"))
}
})
.transpose()
}
fn header_value(headers: &HeaderMap, name: &str) -> S3Result<Option<String>> {
headers
.get(name)
.map(|value| {
value
.to_str()
.map(str::to_owned)
.map_err(|_| s3_error!(InvalidArgument, "invalid header value"))
})
.transpose()
}
#[cfg(test)]
mod tests {
use super::*;
use crate::app::storage_api::s3::dto::EncodingType;
fn uri(value: &str) -> Uri {
value.parse().expect("test URI should parse")
}
#[test]
fn metadata_route_list_objects_v2_input_parses_query_headers_and_decodes_bucket() {
let mut headers = HeaderMap::new();
headers.insert("x-amz-expected-bucket-owner", "123456789012".parse().expect("valid header"));
headers.insert("x-amz-request-payer", "requester".parse().expect("valid header"));
let input = list_objects_v2_input(
"demo bucket".to_string(),
&uri("/demo%20bucket?list-type=2&metadata=true&prefix=logs%2F&delimiter=%2F&encoding-type=url&fetch-owner=true&max-keys=25&continuation-token=opaque&start-after=start"),
&headers,
)
.expect("list objects v2 input should parse");
assert_eq!(input.bucket, "demo bucket");
assert_eq!(input.prefix.as_deref(), Some("logs/"));
assert_eq!(input.delimiter.as_deref(), Some("/"));
assert_eq!(input.encoding_type.as_ref().map(EncodingType::as_str), Some(EncodingType::URL));
assert_eq!(input.fetch_owner, Some(true));
assert_eq!(input.max_keys, Some(25));
assert_eq!(input.continuation_token.as_deref(), Some("opaque"));
assert_eq!(input.start_after.as_deref(), Some("start"));
assert_eq!(input.expected_bucket_owner.as_deref(), Some("123456789012"));
assert_eq!(input.request_payer.as_ref().map(|payer| payer.as_str()), Some("requester"));
}
#[test]
fn metadata_route_list_object_versions_input_parses_markers_and_rejects_invalid_encoding() {
let input = list_object_versions_input(
"bucket".to_string(),
&uri("/bucket?versions&metadata=true&prefix=logs%2F&delimiter=%2F&encoding-type=url&key-marker=start&version-id-marker=v1&max-keys=10"),
&HeaderMap::new(),
)
.expect("list versions input should parse");
assert_eq!(input.bucket, "bucket");
assert_eq!(input.prefix.as_deref(), Some("logs/"));
assert_eq!(input.delimiter.as_deref(), Some("/"));
assert_eq!(input.encoding_type.as_ref().map(EncodingType::as_str), Some(EncodingType::URL));
assert_eq!(input.key_marker.as_deref(), Some("start"));
assert_eq!(input.version_id_marker.as_deref(), Some("v1"));
assert_eq!(input.max_keys, Some(10));
let err = list_object_versions_input(
"bucket".to_string(),
&uri("/bucket?versions&metadata=true&encoding-type=xml"),
&HeaderMap::new(),
)
.expect_err("invalid encoding-type should be rejected");
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InvalidArgument);
}
#[test]
fn metadata_route_listing_parameters_refuse_malformed_values() {
for target in [
"/b?list-type=2&metadata=true&max-keys=ten",
"/b?list-type=2&metadata=true&max-keys=99999999999",
"/b?list-type=2&metadata=true&fetch-owner=yes",
"/b?list-type=2&metadata=true&encoding-type=URL",
"/b?list-type=2&metadata=true&encoding-type=",
] {
let err = list_objects_v2_input("b".to_owned(), &uri(target), &HeaderMap::new()).expect_err(target);
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InvalidArgument, "{target}");
}
let mut headers = HeaderMap::new();
headers.insert(
"x-amz-request-payer",
http::HeaderValue::from_bytes(b"\xffpayer").expect("an opaque header value"),
);
let err = list_object_versions_input("b".to_owned(), &uri("/b?versions&metadata=true"), &headers)
.expect_err("a non-text header is refused");
assert_eq!(err.code(), &rustfs_s3_types::S3ErrorCode::InvalidArgument);
}
}
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -26,7 +26,7 @@ pub(crate) mod bucket_list_through;
pub mod bucket_usecase;
pub mod context;
pub(crate) mod dto_values;
pub(crate) mod metadata_route;
pub(crate) mod metadata_listing;
pub mod multipart_usecase;
pub mod object;
pub(crate) mod object_data_cache;
@@ -39,6 +39,7 @@ pub(crate) mod select_request;
pub(crate) mod storage_api;
pub(crate) mod table_list_isolation;
pub(crate) mod trailer_adapter;
pub(crate) mod website;
// Request-scoped facts are not AppContext resolution, so consumers import them
// from here rather than through `app::context`.
-1
View File
@@ -132,7 +132,6 @@ mod create;
mod create_values;
pub(crate) use create_values::{CreateMultipartUploadFields, CreateMultipartUploadResult};
mod list;
#[cfg(feature = "gateway-stack")]
pub(crate) use list::ListPartsInfo;
pub(crate) use list::{ListPartsFields, ListPartsTarget};
#[cfg(test)]
-12
View File
@@ -16,13 +16,10 @@
use super::*;
use crate::app::RequestEnvelope;
#[cfg(any(feature = "gateway-stack", test))]
use crate::app::storage_api::multipart_usecase::contract::multipart::{MultipartAbortAuthorization, MultipartOwner};
use http::StatusCode;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, RecordedUpload, UploadIdClaim, resolve_upload, resolve_upload_async};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
#[cfg(any(feature = "gateway-stack", test))]
use std::sync::OnceLock;
pub(crate) enum AbortMultipartUploadTarget {
@@ -31,7 +28,6 @@ pub(crate) enum AbortMultipartUploadTarget {
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
@@ -43,7 +39,6 @@ impl AbortMultipartUploadTarget {
fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
@@ -79,7 +74,6 @@ impl DefaultMultipartUsecase {
let (bucket, key) = target.resource();
let mut opts = ObjectOptions::default();
apply_bucket_generation_guard(&envelope, bucket, &mut opts)?;
#[cfg(any(feature = "gateway-stack", test))]
if matches!(&target, AbortMultipartUploadTarget::Gateway { .. }) && opts.expected_bucket_incarnation_id.is_none()
{
return Err(S3Error::with_message(S3ErrorCode::InternalError, "bucket generation guard is missing"));
@@ -95,7 +89,6 @@ impl DefaultMultipartUsecase {
.abort_multipart_upload(bucket, key, upload_id, &opts)
.await
.map_err(multipart_upload_api_error)?,
#[cfg(any(feature = "gateway-stack", test))]
AbortMultipartUploadTarget::Gateway { bucket, key, upload_id } => {
abort_gateway(&store, bucket, key, upload_id, opts).await?
}
@@ -112,10 +105,8 @@ impl DefaultMultipartUsecase {
}
}
#[cfg(any(feature = "gateway-stack", test))]
struct AbortOwner(MultipartOwner);
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for AbortOwner {
fn bucket(&self) -> &str {
&self.0.bucket
@@ -126,7 +117,6 @@ impl RecordedUpload for AbortOwner {
}
}
#[cfg(any(feature = "gateway-stack", test))]
struct AbortAuthorization {
bucket: BucketName,
key: ObjectKey,
@@ -134,7 +124,6 @@ struct AbortAuthorization {
observed: OnceLock<MultipartOwner>,
}
#[cfg(any(feature = "gateway-stack", test))]
impl MultipartAbortAuthorization for AbortAuthorization {
type Error = StorageError;
@@ -148,7 +137,6 @@ impl MultipartAbortAuthorization for AbortAuthorization {
}
}
#[cfg(any(feature = "gateway-stack", test))]
async fn abort_gateway(
store: &ECStore,
bucket: &BucketName,
@@ -1195,7 +1195,6 @@ mod tests {
let response = result.expect("verified key completes and replays");
assert!(response.checksum_sha256.is_some(), "verified key retains checksum response");
assert_eq!(response.sse_customer_key_md5.as_deref(), Some(customer_md5));
#[cfg(feature = "gateway-stack")]
{
use rustfs_gateway::{Limits, MetaView, OperationCodec, TargetKind, WireRequest};
let request = http::Request::builder()
-9
View File
@@ -19,13 +19,10 @@ use crate::app::RequestEnvelope;
use crate::app::dto_values;
use crate::app::storage_api::multipart_usecase::contract::multipart::ListMultipartsInfo;
pub(crate) use crate::app::storage_api::multipart_usecase::contract::multipart::ListPartsInfo;
#[cfg(any(feature = "gateway-stack", test))]
use crate::app::storage_api::multipart_usecase::contract::multipart::ListPartsOwner;
use rustfs_gateway_types::dto as gateway_dto;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, RecordedUpload, UploadIdClaim, resolve_upload_async};
use rustfs_s3_types::S3Result;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_s3_types::{S3Error, S3ErrorCode};
pub(crate) enum ListPartsTarget {
@@ -34,7 +31,6 @@ pub(crate) enum ListPartsTarget {
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
@@ -46,7 +42,6 @@ impl ListPartsTarget {
fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
@@ -58,13 +53,11 @@ pub(crate) struct ListPartsFields {
pub(crate) max_parts: Option<i32>,
}
#[cfg(any(feature = "gateway-stack", test))]
struct RecordedListParts {
owner: ListPartsOwner,
page: ListPartsInfo,
}
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for RecordedListParts {
fn bucket(&self) -> &str {
&self.owner.bucket
@@ -75,7 +68,6 @@ impl RecordedUpload for RecordedListParts {
}
}
#[cfg(any(feature = "gateway-stack", test))]
async fn resolve_list_parts<'a, F>(
upload_id: &'a UploadIdClaim,
bucket: &BucketName,
@@ -210,7 +202,6 @@ impl DefaultMultipartUsecase {
.list_object_parts(bucket, key, upload_id, params.part_number_marker, params.max_parts, &opts)
.await
.map_err(|error| multipart_upload_api_error(error).into()),
#[cfg(any(feature = "gateway-stack", test))]
ListPartsTarget::Gateway {
bucket: bucket_name,
key: object_key,
@@ -20,21 +20,17 @@ use crate::app::object::BodyChunks;
use crate::app::object::request_body::{DemandReader, UploadChunks, guard_native_upload_progress};
use crate::app::storage_api::multipart_usecase::access::AuthorizationRequest;
use crate::app::storage_api::multipart_usecase::contract::multipart::MultipartInfo;
#[cfg(any(feature = "gateway-stack", test))]
use crate::app::storage_api::multipart_usecase::contract::multipart::MultipartOwner;
use http_body::Body as _;
use rustfs_gateway_stream::Body;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{RecordedUpload, resolve_upload_async};
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
#[cfg(any(feature = "gateway-stack", test))]
pub(super) struct PartUpload {
pub(super) owner: MultipartOwner,
pub(super) info: MultipartInfo,
}
#[cfg(any(feature = "gateway-stack", test))]
impl RecordedUpload for PartUpload {
fn bucket(&self) -> &str {
&self.owner.bucket
@@ -50,7 +46,6 @@ async fn lookup_part_upload(store: &ECStore, bucket: &str, key: &str, id: &str,
result.map_err(|error| multipart_upload_api_error(error).into())
}
#[cfg(any(feature = "gateway-stack", test))]
pub(super) async fn lookup_recorded_part_upload(
store: &ECStore,
bucket: &str,
@@ -221,7 +216,6 @@ impl DefaultMultipartUsecase {
let info = lookup_part_upload(&store, bucket, key, upload_id, &opts).await?;
(upload_id.clone(), info)
}
#[cfg(any(feature = "gateway-stack", test))]
UploadPartTarget::Gateway {
bucket: bucket_name,
key: object_key,
@@ -269,7 +269,6 @@ impl DefaultMultipartUsecase {
.map_err(multipart_upload_api_error)?;
(upload_id, info)
}
#[cfg(any(feature = "gateway-stack", test))]
UploadPartTarget::Gateway {
bucket: bound_bucket,
key: bound_key,
@@ -16,7 +16,6 @@
use crate::app::object::FrameTrailers;
use rustfs_gateway_stream::Body;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::{BucketName, ObjectKey, UploadIdClaim};
use rustfs_rio::SharedTrailerSource;
use std::sync::Arc;
@@ -27,7 +26,6 @@ pub(crate) enum UploadPartTarget {
key: String,
upload_id: String,
},
#[cfg(any(feature = "gateway-stack", test))]
Gateway {
bucket: BucketName,
key: ObjectKey,
@@ -39,7 +37,6 @@ impl UploadPartTarget {
pub(super) fn resource(&self) -> (&str, &str) {
match self {
Self::Legacy { bucket, key, .. } => (bucket, key),
#[cfg(any(feature = "gateway-stack", test))]
Self::Gateway { bucket, key, .. } => (bucket.as_str(), key.as_str()),
}
}
@@ -19,99 +19,98 @@ use http_body_util::{BodyExt as _, Empty};
use hyper_util::rt::TokioIo;
fn get_object_attributes_completion(stack: Option<&'static str>) {
temp_env::with_vars([("RUSTFS_S3_STACK", stack), ("RUSTFS_AUDIT_ENABLE", Some("true"))], || {
crate::app::gating_test_env::run_large_stack_test("attributes-completion", || async {
use crate::app::storage_api::object_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::test::contract::object::ObjectIO as _;
use crate::embedded::{RustFSServerBuilder, find_available_port};
temp_env::with_vars(
[
(crate::config::RETIRED_S3_STACK_SELECTOR, stack),
("RUSTFS_AUDIT_ENABLE", Some("true")),
],
|| {
crate::app::gating_test_env::run_large_stack_test("attributes-completion", || async {
use crate::app::storage_api::object_usecase::helper::MutationCompletionProbe;
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::test::contract::object::ObjectIO as _;
use crate::embedded::{RustFSServerBuilder, find_available_port};
let server = RustFSServerBuilder::new()
.address(format!("127.0.0.1:{}", find_available_port().unwrap()))
.access_key("attributesprobe")
.secret_key("attributes-fixture-test-secret")
.build()
.await
.unwrap();
crate::server::refresh_audit_module_enabled();
let context = crate::app::runtime_sources::current_app_context().unwrap();
let store = context.object_store();
let bucket = "attributes-completion";
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": ["s3:GetObject", "s3:GetObjectAttributes"],
"Resource": [format!("arn:aws:s3:::{bucket}/object"), format!("arn:aws:s3:::{bucket}/missing")],
}],
});
store
.update_bucket_metadata_config(
bucket,
crate::app::storage_api::object_usecase::bucket::metadata::BUCKET_POLICY_CONFIG,
serde_json::to_vec(&policy).unwrap(),
)
.await
.unwrap();
store
.put_object(
bucket,
"object",
&mut crate::app::storage_api::object_usecase::StoragePutObjReader::from_vec(b"payload".to_vec()),
&Default::default(),
)
.await
.unwrap();
let probe = MutationCompletionProbe::install(bucket);
for (key, expected) in [("object", 200), ("missing", 404), ("other", 403)] {
let socket = tokio::net::TcpStream::connect(server.address()).await.unwrap();
let (mut sender, connection) = hyper::client::conn::http1::handshake(TokioIo::new(socket)).await.unwrap();
let transport = tokio::spawn(connection);
let request = http::Request::get(format!("/{bucket}/{key}?attributes"))
.header(http::header::HOST, "localhost")
.header("x-amz-object-attributes", "ObjectSize")
.body(Empty::<Bytes>::new())
let server = RustFSServerBuilder::new()
.address(format!("127.0.0.1:{}", find_available_port().unwrap()))
.access_key("attributesprobe")
.secret_key("attributes-fixture-test-secret")
.build()
.await
.unwrap();
let response = sender.send_request(request).await.unwrap();
assert_eq!(response.status().as_u16(), expected);
response.into_body().collect().await.unwrap();
drop(sender);
transport.abort();
let _ = transport.await;
}
probe.wait_for(2, 0).await;
let audits = probe
.audits
.lock()
.unwrap()
.iter()
.filter(|entry| entry.api.name.as_deref() == Some(rustfs_s3_ops::S3Operation::GetObjectAttributes.as_str()))
.map(|entry| (entry.api.object.clone(), entry.api.status.clone(), entry.api.status_code))
.collect::<Vec<_>>();
assert_eq!(
audits,
[
(Some("object".to_owned()), Some("success".to_owned()), Some(200)),
(Some("missing".to_owned()), Some("failure".to_owned()), Some(500)),
],
);
assert!(probe.events.lock().unwrap().is_empty());
server.shutdown().await;
})
})
crate::server::refresh_audit_module_enabled();
let context = crate::app::runtime_sources::current_app_context().unwrap();
let store = context.object_store();
let bucket = "attributes-completion";
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "*"},
"Action": ["s3:GetObject", "s3:GetObjectAttributes"],
"Resource": [format!("arn:aws:s3:::{bucket}/object"), format!("arn:aws:s3:::{bucket}/missing")],
}],
});
store
.update_bucket_metadata_config(
bucket,
crate::app::storage_api::object_usecase::bucket::metadata::BUCKET_POLICY_CONFIG,
serde_json::to_vec(&policy).unwrap(),
)
.await
.unwrap();
store
.put_object(
bucket,
"object",
&mut crate::app::storage_api::object_usecase::StoragePutObjReader::from_vec(b"payload".to_vec()),
&Default::default(),
)
.await
.unwrap();
let probe = MutationCompletionProbe::install(bucket);
for (key, expected) in [("object", 200), ("missing", 404), ("other", 403)] {
let socket = tokio::net::TcpStream::connect(server.address()).await.unwrap();
let (mut sender, connection) = hyper::client::conn::http1::handshake(TokioIo::new(socket)).await.unwrap();
let transport = tokio::spawn(connection);
let request = http::Request::get(format!("/{bucket}/{key}?attributes"))
.header(http::header::HOST, "localhost")
.header("x-amz-object-attributes", "ObjectSize")
.body(Empty::<Bytes>::new())
.unwrap();
let response = sender.send_request(request).await.unwrap();
assert_eq!(response.status().as_u16(), expected);
response.into_body().collect().await.unwrap();
drop(sender);
transport.abort();
let _ = transport.await;
}
probe.wait_for(2, 0).await;
let audits = probe
.audits
.lock()
.unwrap()
.iter()
.filter(|entry| entry.api.name.as_deref() == Some(rustfs_s3_ops::S3Operation::GetObjectAttributes.as_str()))
.map(|entry| (entry.api.object.clone(), entry.api.status.clone(), entry.api.status_code))
.collect::<Vec<_>>();
assert_eq!(
audits,
[
(Some("object".to_owned()), Some("success".to_owned()), Some(200)),
(Some("missing".to_owned()), Some("failure".to_owned()), Some(500)),
],
);
assert!(probe.events.lock().unwrap().is_empty());
server.shutdown().await;
})
},
)
}
#[test]
#[serial_test::serial]
fn legacy_get_object_attributes_completion() {
get_object_attributes_completion(None);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_get_object_attributes_completion() {
get_object_attributes_completion(Some("gateway"));
get_object_attributes_completion(None);
}
-1
View File
@@ -1188,7 +1188,6 @@ impl DefaultObjectUsecase {
/// The access phase has installed `ReqInfo` and the pending bucket-generation guard; this
/// adds `ReplicateDelete` for a replica delete, repeats `ForceDeleteObject` for a non-owner,
/// and authorizes every descendant of a recursive force delete.
#[cfg(any(feature = "gateway-stack", test))]
#[instrument(level = "info", skip(self, envelope, input))]
pub async fn execute_delete_object(
&self,
-6
View File
@@ -2018,7 +2018,6 @@ impl DefaultObjectUsecase {
if envelope.post_object().is_some() && !envelope.is_gateway_post() {
return Err(rustfs_s3_types::s3_error!(NotImplemented, "POST archive extraction is not supported"));
}
#[cfg(feature = "gateway-stack")]
let input = if envelope.is_gateway_post() {
let mut input = input;
let archive = input
@@ -2045,10 +2044,6 @@ impl DefaultObjectUsecase {
}
input
};
#[cfg(not(feature = "gateway-stack"))]
if envelope.is_gateway_post() {
return Err(rustfs_s3_types::s3_error!(NotImplemented, "POST archive extraction is not supported"));
}
Box::pin(self.execute_put_object_extract_inner(envelope, input)).await
}
@@ -2831,7 +2826,6 @@ impl DefaultObjectUsecase {
}
}
#[cfg(feature = "gateway-stack")]
#[path = "extract/post_archive_preflight.rs"]
pub(super) mod post_archive_preflight;
-1
View File
@@ -4502,7 +4502,6 @@ impl DefaultObjectUsecase {
crate::app::test_seams::get_object(self, request).await
}
#[cfg(feature = "gateway-stack")]
pub(crate) async fn execute_get_object(
&self,
envelope: RequestEnvelope,
+2 -9
View File
@@ -86,23 +86,16 @@ fn operation_counts() -> std::collections::BTreeMap<&'static str, u64> {
.collect()
}
#[test]
#[serial_test::serial]
fn get_boundary_legacy() {
get_boundary(None);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn get_boundary_gateway() {
get_boundary(Some("gateway"));
get_boundary(None);
}
fn get_boundary(stack: Option<&str>) {
temp_env::with_vars(
[
("RUSTFS_S3_STACK", stack),
(crate::config::RETIRED_S3_STACK_SELECTOR, stack),
("RUSTFS_AUDIT_ENABLE", Some("true")),
("RUSTFS_SSE_C_REQUIRE_TLS", Some("false")),
],
-2
View File
@@ -21,7 +21,6 @@ use crate::app::dto_values::{stored_etag, timestamp};
use crate::app::storage_api::s3::header::X_AMZ_RESTORE;
use crate::app::{Answered, RequestEnvelope};
use rustfs_gateway_types::ByteRange;
#[cfg(feature = "gateway-stack")]
use rustfs_gateway_types::RangeParse;
use rustfs_gateway_types::dto as gateway_dto;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result, s3_error};
@@ -447,7 +446,6 @@ impl DefaultObjectUsecase {
Ok(response)
}
#[cfg(feature = "gateway-stack")]
pub(crate) async fn execute_head_object(
&self,
envelope: RequestEnvelope,
+6 -28
View File
@@ -214,7 +214,7 @@ const INVALID_HEAD_ETAGS: &[HeadEtagRow] = &[
fn head_etag_wire_and_audit(stack: Option<&'static str>, rows: &'static [HeadEtagRow]) {
temp_env::with_vars(
[
("RUSTFS_S3_STACK", stack),
(crate::config::RETIRED_S3_STACK_SELECTOR, stack),
("RUSTFS_AUDIT_ENABLE", Some("true")),
("RUSTFS_NOTIFY_ENABLE", Some("false")),
],
@@ -307,28 +307,14 @@ fn head_etag_wire_and_audit(stack: Option<&'static str>, rows: &'static [HeadEta
#[test]
#[serial_test::serial]
fn legacy_head_etag_wire_preserves_typed_values_and_audit() {
fn gateway_head_etag_wire_preserves_typed_values_and_audit() {
head_etag_wire_and_audit(None, VALID_HEAD_ETAGS);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_head_etag_wire_preserves_typed_values_and_audit() {
head_etag_wire_and_audit(Some("gateway"), VALID_HEAD_ETAGS);
}
#[test]
#[serial_test::serial]
fn legacy_head_invalid_etag_is_rejected_after_audit() {
head_etag_wire_and_audit(None, INVALID_HEAD_ETAGS);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_head_invalid_etag_is_rejected_after_audit() {
head_etag_wire_and_audit(Some("gateway"), INVALID_HEAD_ETAGS);
head_etag_wire_and_audit(None, INVALID_HEAD_ETAGS);
}
async fn head_test_server() -> crate::embedded::RustFSServer {
@@ -372,11 +358,10 @@ async fn head_wire_response(
http::Response::from_parts(parts, body)
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_head_authorization_uses_the_request_bound_context() {
temp_env::with_var("RUSTFS_S3_STACK", Some("gateway"), || {
temp_env::with_var(crate::config::RETIRED_S3_STACK_SELECTOR, None::<&str>, || {
crate::app::gating_test_env::run_large_stack_test("head-gateway-authorization", || async {
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::test::contract::object::ObjectIO as _;
@@ -430,7 +415,7 @@ fn gateway_head_authorization_uses_the_request_bound_context() {
}
fn head_conditional_headers_preserve_stored_facts(stack: Option<&'static str>) {
temp_env::with_var("RUSTFS_S3_STACK", stack, || {
temp_env::with_var(crate::config::RETIRED_S3_STACK_SELECTOR, stack, || {
crate::app::gating_test_env::run_large_stack_test("head-conditional-headers", || async {
use crate::app::storage_api::test::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::test::contract::object::{ObjectIO as _, ObjectOperations as _};
@@ -504,13 +489,6 @@ fn head_conditional_headers_preserve_stored_facts(stack: Option<&'static str>) {
#[test]
#[serial_test::serial]
fn legacy_head_conditional_headers_preserve_stored_facts() {
fn gateway_head_conditional_headers_preserve_stored_facts() {
head_conditional_headers_preserve_stored_facts(None);
}
#[cfg(feature = "gateway-stack")]
#[test]
#[serial_test::serial]
fn gateway_head_conditional_headers_preserve_stored_facts() {
head_conditional_headers_preserve_stored_facts(Some("gateway"));
}
-3
View File
@@ -230,15 +230,12 @@ mod get;
mod get_response;
pub(crate) use get_response::{GetObjectBody, GetObjectFields, GetObjectOutputFacts};
mod head;
#[cfg(feature = "gateway-stack")]
pub(crate) use head::ResponseEtag;
pub(crate) use head::{HeadObjectFields, HeadObjectOutputFacts};
mod internal_put;
mod native_body;
mod on_demand_migration_put;
#[cfg(feature = "gateway-stack")]
pub(crate) mod post_body;
#[cfg(feature = "gateway-stack")]
pub(crate) mod post_headers;
mod put;
mod put_input;
-1
View File
@@ -4384,7 +4384,6 @@ mod tests {
}
#[tokio::test]
#[cfg(feature = "gateway-stack")]
async fn gateway_post_archive_is_refused_before_body_on_both_internal_paths() {
for direct_extract in [false, true] {
let (body, polls) = PollCountingBody::streaming_blob();
-2
View File
@@ -25,7 +25,6 @@ pub(crate) struct PutObjectWriteInput {
pub(crate) bucket: String,
pub(crate) key: String,
pub(crate) body: Option<Body>,
#[cfg(feature = "gateway-stack")]
pub(crate) post_archive: Option<super::post_body::PreflightedPostArchive>,
pub(crate) frame_trailers: Option<std::sync::Arc<super::native_body::FrameTrailers>>,
pub(crate) trailing_headers: Option<SharedTrailerSource>,
@@ -83,7 +82,6 @@ pub(crate) struct PutObjectWriteOutput {
impl PutObjectWriteInput {
/// Gateway frames publish to the same source that checksum readers observe.
#[cfg(feature = "gateway-stack")]
pub(crate) fn capture_frame_trailers(&mut self) {
let trailers = std::sync::Arc::new(super::native_body::FrameTrailers::default());
self.frame_trailers = Some(trailers.clone());
-3
View File
@@ -27,7 +27,6 @@ use crate::app::storage_api::object_usecase::helper::OperationHelper;
use crate::app::storage_api::object_usecase::options::get_opts;
use crate::error::ApiError;
use http::{HeaderMap, StatusCode};
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::{GetObjectAclInput, PutObjectAclInput};
use rustfs_gateway_types::dto::{GetObjectAclOutput, Grant, Grantee, Permission, PutObjectAclOutput, Type};
use rustfs_s3_ops::S3Operation;
@@ -40,7 +39,6 @@ impl ObjectMetaUsecase {
///
/// `InternalError` before the store is up; the store's refusal (`NoSuchKey`, `NoSuchVersion`,
/// `NoSuchBucket`) when the object does not exist.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_get_object_acl(
&self,
envelope: RequestEnvelope,
@@ -72,7 +70,6 @@ impl ObjectMetaUsecase {
///
/// `MethodNotAllowed` for a Veeam SOSAPI system object; `InternalError` before the store is up;
/// the store's refusal when the object does not exist; `NotImplemented` for XML grants.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_put_object_acl(
&self,
envelope: RequestEnvelope,
-2
View File
@@ -34,7 +34,6 @@ use crate::app::storage_api::object_usecase::helper::OperationHelper;
use crate::app::storage_api::object_usecase::options::get_opts;
use crate::app::storage_api::object_usecase::{StorageObjectOptions as ObjectOptions, parse_object_lock_legal_hold_native};
use crate::error::ApiError;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::GetObjectLegalHoldInput;
use rustfs_gateway_types::dto::{GetObjectLegalHoldOutput, ObjectLockLegalHold, Status};
use rustfs_gateway_types::dto::{PutObjectLegalHoldOutput, RequestCharged};
@@ -158,7 +157,6 @@ impl ObjectMetaUsecase {
/// `MethodNotAllowed` for a Veeam SOSAPI system object; `InternalError` before the store is up
/// or when the object cannot be read; `NoSuchBucket`; the Object Lock refusal of a bucket
/// without Object Lock.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_get_object_legal_hold(
&self,
envelope: RequestEnvelope,
-2
View File
@@ -37,7 +37,6 @@ use crate::app::storage_api::object_usecase::helper::OperationHelper;
use crate::app::storage_api::object_usecase::options::get_opts;
use crate::app::storage_api::object_usecase::{StorageObjectOptions as ObjectOptions, parse_object_lock_retention_native};
use crate::error::ApiError;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::GetObjectRetentionInput;
use rustfs_gateway_types::dto::{GetObjectRetentionOutput, Mode, ObjectLockRetention};
use rustfs_gateway_types::dto::{PutObjectRetentionOutput, RequestCharged};
@@ -178,7 +177,6 @@ impl ObjectMetaUsecase {
/// `MethodNotAllowed` for a Veeam SOSAPI system object; `InternalError` before the store is up,
/// when the object cannot be read, or for a stored date outside the answerable range; the
/// Object Lock refusal of a bucket without Object Lock.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_get_object_retention(
&self,
envelope: RequestEnvelope,
-5
View File
@@ -38,9 +38,7 @@ use crate::error::ApiError;
use http::{HeaderMap, StatusCode};
use metrics::{counter, histogram};
use rustfs_gateway_types::cors_tagging::PersistedTag;
#[cfg(feature = "gateway-stack")]
use rustfs_gateway_types::dto::PutObjectTaggingInput;
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::{DeleteObjectTaggingInput, GetObjectTaggingInput};
use rustfs_gateway_types::dto::{DeleteObjectTaggingOutput, GetObjectTaggingOutput, PutObjectTaggingOutput, Tag};
use rustfs_s3_ops::S3Operation;
@@ -67,7 +65,6 @@ impl ObjectMetaUsecase {
/// `MethodNotAllowed` for a Veeam SOSAPI system object or a named delete marker; `NoSuchKey` for
/// a missing object or a current delete marker; `NoSuchVersion` for a missing named version;
/// `InvalidArgument` for a malformed version id; `InternalError` before the store is up.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_get_object_tagging(
&self,
envelope: RequestEnvelope,
@@ -170,7 +167,6 @@ impl ObjectMetaUsecase {
///
/// Returns `InvalidTag` for an invalid tag set, a refusal for reserved objects, or the
/// storage error when the tags cannot be written.
#[cfg(feature = "gateway-stack")]
pub async fn execute_put_object_tagging(
&self,
envelope: RequestEnvelope,
@@ -296,7 +292,6 @@ impl ObjectMetaUsecase {
///
/// `MethodNotAllowed` for a Veeam SOSAPI system object; `InvalidRequest` for a reserved table
/// catalog object; `InternalError` before the store is up; the store's refusal otherwise.
#[cfg(any(feature = "gateway-stack", test))]
pub async fn execute_delete_object_tagging(
&self,
envelope: RequestEnvelope,
-2
View File
@@ -19,7 +19,6 @@
//! NOT responsible for: authorization (there is none: the answer is the same for every caller).
//! Upstream: both edges. Downstream: none.
#[cfg(any(feature = "gateway-stack", test))]
use rustfs_gateway_types::dto::GetObjectTorrentInput;
use rustfs_gateway_types::dto::GetObjectTorrentOutput;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
@@ -29,7 +28,6 @@ use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
/// # Errors
///
/// Always `NoSuchKey`.
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) fn execute_get_object_torrent(_input: GetObjectTorrentInput) -> S3Result<GetObjectTorrentOutput> {
answer_get_object_torrent()
}
-2
View File
@@ -22,8 +22,6 @@ pub(crate) use crate::runtime_sources::{
current_notification_system_for_context, current_object_data_cache_handle_for_context,
current_object_store_handle_for_context,
};
#[cfg(test)]
pub(crate) use crate::runtime_sources::{IamInterface, KmsInterface};
use rustfs_s3select_api::request::SelectObjectContentInput;
use rustfs_s3select_api::{QueryResult, server::dbms::DatabaseManagerSystem};
use std::sync::Arc;
-1
View File
@@ -68,7 +68,6 @@ pub(crate) fn select_input_from_s3s(
}
/// Moves the gateway's flat request into the shape the query engine reads.
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) fn select_input_from_gateway(
input: rustfs_gateway_types::dto::SelectObjectContentInput,
) -> engine::SelectObjectContentInput {
+2 -6
View File
@@ -126,15 +126,14 @@ pub(crate) mod data_usage {
pub(crate) use crate::storage::storage_api::ecstore_data_usage::quota_object_size;
#[cfg(test)]
pub(crate) async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut rustfs_data_usage::DataUsageInfo) {
crate::storage::storage_api::ecstore_data_usage::apply_bucket_usage_memory_overlay(data_usage_info).await;
}
pub(crate) async fn load_admin_data_usage_with_memory_overlay(
pub(crate) async fn load_admin_data_usage_from_backend_cached(
store: Arc<crate::storage::storage_api::ECStore>,
) -> Result<rustfs_data_usage::DataUsageInfo, crate::storage::storage_api::StorageError> {
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_with_memory_overlay(store).await
crate::storage::storage_api::ecstore_data_usage::load_admin_data_usage_from_backend_cached(store).await
}
pub(crate) async fn load_bucket_usage_from_backend_cached(
@@ -1602,7 +1601,6 @@ pub(crate) mod concurrency {
}
pub(crate) mod compression {
#[cfg(feature = "gateway-stack")]
pub(crate) use crate::storage::storage_api::ecstore_compression::DISK_COMPRESSION_EXCLUDED_CONTENT_TYPES;
pub(crate) use crate::storage::storage_api::ecstore_compression::{
MIN_DISK_COMPRESSIBLE_SIZE, is_disk_compressible, is_multipart_disk_compression_enabled,
@@ -1872,12 +1870,10 @@ pub(crate) mod multipart_usecase {
}
pub(crate) mod multipart {
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) use super::super::super::storage_contracts::multipart::ListPartsOwner;
pub(crate) use super::super::super::storage_contracts::{
CompletePart, ListMultipartsInfo, ListPartsInfo, MultipartInfo, MultipartOperations, MultipartUploadResult,
};
#[cfg(any(feature = "gateway-stack", test))]
pub(crate) use super::super::super::storage_contracts::{MultipartAbortAuthorization, MultipartOwner};
pub(crate) use crate::storage::storage_api::s3_api_consumer::multipart::contract::multipart::MAX_MULTIPART_PART_NUMBER;
}
+557
View File
@@ -0,0 +1,557 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Static website hosting: which requests a website host claims, and how they are answered.
//!
//! Responsible for: [`WebsiteHost`], the one copy of the website rules both S3 stacks serve —
//! the `Host` that selects a bucket, the index and error documents, the redirect rules and the
//! object's own redirect location — read anonymously through the owned GetObject and HeadObject
//! authorization and use cases, marking each GetObject with `WebsiteRead` so it reports the
//! object's redirect location.
//! NOT responsible for: where the stack mounts the host (`server/stack.rs`), or any object read
//! itself (the use cases).
//! Upstream: the gateway stack's website service. Downstream: the
//! owned GetObject and HeadObject authorization and use cases, and the bucket's website
//! configuration.
use crate::app::object_usecase::{DefaultObjectUsecase, GetObjectBody, GetObjectFields, GetObjectOutputFacts, HeadObjectFields};
use crate::app::runtime_sources::ServerContextSlot;
use crate::app::storage_api::s3::dto::{
ETagCondition, GetObjectInput, Range, Timestamp as WireTimestamp, TimestampFormat as WireFormat,
};
use crate::app::{Answered, RequestEnvelope, RequestHead};
use crate::storage::access::access_owned::object_read::{authorize_get_object, authorize_head_object};
use crate::storage::req_info::ReqInfo;
use crate::storage::request_markers::WebsiteRead;
use crate::storage::{get_bucket_website_config_for_store, validate_website_configuration};
use http::header::{CONTENT_TYPE, HOST, HeaderName};
use http::{Extensions, HeaderMap, HeaderValue, Method, Response, StatusCode, Uri};
use rustfs_gateway_stream::Body;
use rustfs_gateway_types::TimestampFormat;
use rustfs_gateway_types::persistence::{PersistedErrorDocument, PersistedRoutingRule as RoutingRule};
use rustfs_io_metrics::record_s3_op;
use rustfs_s3_ops::S3Operation;
use rustfs_s3_types::{S3Error, S3ErrorCode, S3Result};
use std::sync::Arc;
/// An answer of the website host.
pub(crate) type WebsiteResponse = Response<Body>;
/// The headers a website read drops before a lookup that must not be conditional or ranged.
const CONDITIONAL_HEADERS: [HeaderName; 5] = [
http::header::RANGE,
http::header::IF_MATCH,
http::header::IF_NONE_MATCH,
http::header::IF_MODIFIED_SINCE,
http::header::IF_UNMODIFIED_SINCE,
];
/// The website host of one server: the configured website domains, the scheme its redirects
/// default to, and the server whose buckets it serves.
pub(crate) struct WebsiteHost {
domains: Vec<String>,
scheme: &'static str,
server_ctx: Arc<ServerContextSlot>,
}
impl WebsiteHost {
/// The host for `domains`, or `None` when no website domain is configured.
pub(crate) fn new(domains: Vec<String>, scheme: &'static str, server_ctx: Arc<ServerContextSlot>) -> Option<Self> {
(!domains.is_empty()).then_some(Self {
domains,
scheme,
server_ctx,
})
}
/// Whether the request's `Host` is a website domain, so this host answers it.
pub(crate) fn claims(&self, uri: &Uri, headers: &HeaderMap) -> bool {
website_target(uri, headers, &self.domains).is_some()
}
/// Answers one claimed request anonymously. Every failure is a website error page with the
/// failure's status, never an S3 error document.
pub(crate) async fn answer(&self, method: Method, uri: Uri, headers: HeaderMap, extensions: Extensions) -> WebsiteResponse {
let head = method == Method::HEAD;
let Some(target) = website_target(&uri, &headers, &self.domains) else {
return website_error(StatusCode::NOT_FOUND, head);
};
let request = WebsiteRequest {
method,
uri,
headers,
extensions,
server_ctx: Arc::clone(&self.server_ctx),
};
// The reads are large futures; boxed, a website request does not grow the caller's stack.
match Box::pin(serve(&request, target, self.scheme)).await {
Ok(response) => response,
Err(err) => website_error(err.status_code().unwrap_or(StatusCode::INTERNAL_SERVER_ERROR), head),
}
}
}
#[derive(Clone, Debug)]
struct WebsiteTarget {
bucket: String,
key: String,
trailing_slash: bool,
invalid_path: bool,
}
fn website_target(uri: &Uri, headers: &HeaderMap, domains: &[String]) -> Option<WebsiteTarget> {
if domains.is_empty() {
return None;
}
let host_header = headers.get(HOST)?.to_str().ok()?;
let authority = host_header.parse::<http::uri::Authority>().ok()?;
let request_host = authority.host().trim_end_matches('.').to_ascii_lowercase();
let bucket = domains.iter().find_map(|domain| {
if request_host == *domain {
return Some(String::new());
}
request_host.strip_suffix(&format!(".{domain}")).map(str::to_owned)
})?;
let raw_path = uri.path();
let decoded = urlencoding::decode(raw_path);
let invalid_path = decoded.is_err();
let decoded = decoded.unwrap_or_default();
let path = decoded.trim_start_matches('/');
Some(WebsiteTarget {
bucket,
key: path.to_owned(),
trailing_slash: path.is_empty() || path.ends_with('/'),
invalid_path,
})
}
/// One website request as the reads it makes see it.
struct WebsiteRequest {
method: Method,
uri: Uri,
headers: HeaderMap,
extensions: Extensions,
server_ctx: Arc<ServerContextSlot>,
}
impl WebsiteRequest {
fn head(&self) -> bool {
self.method == Method::HEAD
}
/// An anonymous read of `bucket/key`: no credentials and no owner, whatever the request carried.
fn envelope(&self, method: Method, headers: HeaderMap, bucket: &str, key: &str, website_read: bool) -> RequestEnvelope {
let mut extensions = self.extensions.clone();
extensions.insert(Arc::clone(&self.server_ctx));
let mut req_info = extensions.remove::<ReqInfo>().unwrap_or_default();
req_info.cred = None;
req_info.is_owner = false;
req_info.bucket = Some(bucket.to_owned());
req_info.object = Some(key.to_owned());
req_info.version_id = None;
extensions.insert(req_info);
if website_read {
extensions.insert(WebsiteRead);
}
let head = RequestHead {
method,
uri: self.uri.clone(),
headers,
credentials: None,
region: None,
service: None,
};
RequestEnvelope::from_parts(head, extensions)
}
fn usecase(&self) -> DefaultObjectUsecase {
DefaultObjectUsecase::with_context(self.server_ctx.app_context())
}
/// GetObject of `bucket/key`, with the request's range and conditions when `apply_conditions`.
async fn get_object(&self, bucket: &str, key: &str, apply_conditions: bool) -> S3Result<Answered<GetObjectOutputFacts>> {
let (range, headers) = if apply_conditions {
(validated_conditions(&self.headers)?, self.headers.clone())
} else {
(None, without_conditions(&self.headers))
};
let mut envelope = self.envelope(Method::GET, headers, bucket, key, true);
authorize_get_object(&self.server_ctx, &mut envelope, bucket, key, None).await?;
record_s3_op(S3Operation::GetObject);
let fields = GetObjectFields::from_s3s(GetObjectInput {
bucket: bucket.to_owned(),
key: key.to_owned(),
range,
..Default::default()
});
Box::pin(self.usecase().execute_get_object_fields(envelope, fields)).await
}
/// The redirect location HeadObject reports for `bucket/key`, read unconditionally.
async fn head_redirect(&self, bucket: &str, key: &str) -> S3Result<Option<String>> {
let mut envelope = self.envelope(Method::HEAD, without_conditions(&self.headers), bucket, key, false);
authorize_head_object(&self.server_ctx, &mut envelope, bucket, key, None).await?;
record_s3_op(S3Operation::HeadObject);
let fields = HeadObjectFields {
bucket: bucket.to_owned(),
key: key.to_owned(),
version_id: None,
part_number: None,
range: None,
sse_customer_key: None,
sse_customer_key_md5: None,
};
let answered = Box::pin(self.usecase().execute_head_object_fields(envelope, fields)).await?;
Ok(answered.output.dto.website_redirect_location)
}
}
fn header_text(headers: &HeaderMap, name: HeaderName) -> S3Result<Option<String>> {
headers
.get(name)
.map(|value| {
value
.to_str()
.map(str::to_owned)
.map_err(|_| S3Error::new(S3ErrorCode::InvalidArgument))
})
.transpose()
}
/// The range a conditional read asks for, after refusing a malformed range or condition the
/// way the S3 GetObject decoder refuses it.
fn validated_conditions(headers: &HeaderMap) -> S3Result<Option<Range>> {
let range = header_text(headers, http::header::RANGE)?
.map(|value| Range::parse(&value).map_err(|_| S3Error::new(S3ErrorCode::InvalidRange)))
.transpose()?;
for name in [http::header::IF_MATCH, http::header::IF_NONE_MATCH] {
if let Some(value) = headers.get(name) {
ETagCondition::parse_http_header(value.as_bytes()).map_err(|_| S3Error::new(S3ErrorCode::InvalidArgument))?;
}
}
for name in [http::header::IF_MODIFIED_SINCE, http::header::IF_UNMODIFIED_SINCE] {
if let Some(value) = header_text(headers, name)? {
WireTimestamp::parse(WireFormat::HttpDate, &value).map_err(|_| S3Error::new(S3ErrorCode::InvalidArgument))?;
}
}
Ok(range)
}
fn without_conditions(headers: &HeaderMap) -> HeaderMap {
let mut headers = headers.clone();
for name in CONDITIONAL_HEADERS {
headers.remove(name);
}
headers
}
async fn serve(req: &WebsiteRequest, target: WebsiteTarget, website_scheme: &'static str) -> S3Result<WebsiteResponse> {
if !matches!(req.method, Method::GET | Method::HEAD) {
return Ok(website_error(StatusCode::METHOD_NOT_ALLOWED, req.head()));
}
if target.invalid_path {
return Ok(website_error(StatusCode::BAD_REQUEST, req.head()));
}
if target.bucket.is_empty() || target.bucket.starts_with('.') || target.bucket.ends_with('.') || target.bucket.contains("..")
{
return Ok(website_error(StatusCode::NOT_FOUND, req.head()));
}
let store = req
.server_ctx
.installed_object_store()
.ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "website object store unavailable"))?;
let config = match get_bucket_website_config_for_store(&store, &target.bucket).await {
Ok(config) => config,
Err(crate::storage::StorageError::ConfigNotFound | crate::storage::StorageError::BucketNotFound(_)) => {
return Ok(website_error(StatusCode::NOT_FOUND, req.head()));
}
Err(err) => return Err(crate::error::ApiError::from(err).into()),
};
if validate_website_configuration(&config).is_err() {
return Ok(website_error(StatusCode::INTERNAL_SERVER_ERROR, req.head()));
}
if let Some(redirect) = config.redirect_all_requests_to.as_ref() {
return website_redirect(
&req.uri,
&req.headers,
redirect.host_name.as_str(),
redirect.protocol.as_deref(),
&target.key,
StatusCode::MOVED_PERMANENTLY,
website_scheme,
);
}
let index_suffix = config
.index_document
.as_ref()
.map(|doc| doc.suffix.as_str())
.unwrap_or_default();
let key = if target.key.is_empty() || target.trailing_slash {
format!("{}{index_suffix}", target.key)
} else {
target.key.clone()
};
let rules = config.routing_rules.as_deref().unwrap_or_default();
// A rule requiring a status cannot be evaluated until the object lookup completes.
for rule in rules {
if rule
.condition
.as_ref()
.and_then(|condition| condition.http_error_code_returned_equals.as_ref())
.is_some()
{
break;
}
if website_rule_matches(rule, &target.key, StatusCode::OK) {
return website_rule_redirect(req, rule, &target.key, website_scheme);
}
}
let response = req.get_object(&target.bucket, &key, true).await;
if let Err(err) = &response
&& matches!(
err.code(),
S3ErrorCode::NotModified | S3ErrorCode::PreconditionFailed | S3ErrorCode::InvalidRange
)
&& let Ok(Some(location)) = req.head_redirect(&target.bucket, &key).await
{
return website_location(location, StatusCode::MOVED_PERMANENTLY);
}
let status = match &response {
Ok(_) => StatusCode::OK,
Err(err) => err.status_code().unwrap_or(StatusCode::INTERNAL_SERVER_ERROR),
};
if status == StatusCode::NOT_FOUND && !target.trailing_slash && !target.key.is_empty() {
let index_key = format!("{}/{index_suffix}", target.key);
match req.get_object(&target.bucket, &index_key, false).await {
Ok(_) => return website_relative_redirect(&req.uri, &format!("/{}/", target.key), StatusCode::FOUND),
Err(err)
if matches!(
err.code(),
S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion | S3ErrorCode::AccessDenied
) => {}
Err(err) => return Err(err),
}
}
if let Some(rule) = rules.iter().find(|rule| website_rule_matches(rule, &target.key, status)) {
return website_rule_redirect(req, rule, &target.key, website_scheme);
}
match response {
Ok(response) => {
if let Some(location) = response.output.dto.website_redirect_location.clone() {
return website_location(location, StatusCode::MOVED_PERMANENTLY);
}
response_to_website(response, req.head())
}
Err(err) if matches!(err.code(), S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion) => {
website_error_with_document(req, &target.bucket, config.error_document.as_ref(), StatusCode::NOT_FOUND).await
}
Err(err) if err.code() == &S3ErrorCode::NotModified => Ok(website_not_modified(&err)),
Err(_) if status.is_client_error() => {
website_error_with_document(req, &target.bucket, config.error_document.as_ref(), status).await
}
Err(err) => Err(err),
}
}
/// The object's page: its representation headers, and its body unless the request is `HEAD`.
fn response_to_website(response: Answered<GetObjectOutputFacts>, head: bool) -> S3Result<WebsiteResponse> {
let Answered { output, mut headers, .. } = response;
let GetObjectOutputFacts { dto, etag, body } = output;
insert_website_header(&mut headers, http::header::CONTENT_TYPE, dto.content_type.as_deref())?;
insert_website_header(
&mut headers,
http::header::CONTENT_LENGTH,
dto.content_length.map(|length| length.to_string()).as_deref(),
)?;
insert_website_header(&mut headers, http::header::CONTENT_RANGE, dto.content_range.as_deref())?;
let etag = etag.map(|tag| {
if tag.weak {
format!("W/\"{}\"", tag.opaque)
} else {
format!("\"{}\"", tag.opaque)
}
});
insert_website_header(&mut headers, http::header::ETAG, etag.as_deref())?;
insert_website_header(&mut headers, http::header::CACHE_CONTROL, dto.cache_control.as_deref())?;
insert_website_header(&mut headers, http::header::CONTENT_ENCODING, dto.content_encoding.as_deref())?;
insert_website_header(&mut headers, http::header::CONTENT_LANGUAGE, dto.content_language.as_deref())?;
insert_website_header(&mut headers, http::header::CONTENT_DISPOSITION, dto.content_disposition.as_deref())?;
let last_modified = dto
.last_modified
.map(|at| at.render(TimestampFormat::HttpDate).map_err(S3Error::internal_error))
.transpose()?;
insert_website_header(&mut headers, http::header::LAST_MODIFIED, last_modified.as_deref())?;
let body = match body {
_ if head => Body::empty(),
None => Body::empty(),
Some(GetObjectBody::Bytes(bytes)) => Body::from_bytes(bytes),
Some(GetObjectBody::Stream(stream)) => Body::from_stream(stream).map_err(S3Error::internal_error)?,
};
let status = if dto.content_range.is_some() {
StatusCode::PARTIAL_CONTENT
} else {
StatusCode::OK
};
Ok(answer(status, headers, body))
}
fn answer(status: StatusCode, headers: HeaderMap, body: Body) -> WebsiteResponse {
let mut response = Response::new(body);
*response.status_mut() = status;
*response.headers_mut() = headers;
response
}
fn insert_website_header(headers: &mut HeaderMap, name: HeaderName, value: Option<&str>) -> S3Result<()> {
if let Some(value) = value {
headers.insert(name, HeaderValue::from_str(value).map_err(S3Error::internal_error)?);
}
Ok(())
}
async fn website_error_with_document(
req: &WebsiteRequest,
bucket: &str,
error_document: Option<&PersistedErrorDocument>,
status: StatusCode,
) -> S3Result<WebsiteResponse> {
let Some(error_document) = error_document else {
return Ok(website_error(status, req.head()));
};
match req.get_object(bucket, error_document.key.as_str(), false).await {
Ok(response) => {
let mut output = response_to_website(response, req.head())?;
*output.status_mut() = status;
Ok(output)
}
Err(err)
if matches!(
err.code(),
S3ErrorCode::NoSuchKey | S3ErrorCode::NoSuchVersion | S3ErrorCode::AccessDenied
) =>
{
Ok(website_error(status, req.head()))
}
Err(err) => Err(err),
}
}
fn website_error(status: StatusCode, head: bool) -> WebsiteResponse {
let message = status.canonical_reason().unwrap_or("Error");
let body = format!("<html><body>{message}</body></html>");
let mut headers = HeaderMap::new();
headers.insert(CONTENT_TYPE, HeaderValue::from_static("text/html; charset=utf-8"));
answer(status, headers, if head { Body::empty() } else { Body::from_bytes(body.into()) })
}
fn website_not_modified(error: &S3Error) -> WebsiteResponse {
let mut headers = error.headers().cloned().unwrap_or_default();
headers.remove(http::header::CONTENT_LENGTH);
headers.remove(http::header::CONTENT_TYPE);
headers.remove(http::header::TRANSFER_ENCODING);
answer(StatusCode::NOT_MODIFIED, headers, Body::empty())
}
fn website_location(location: String, status: StatusCode) -> S3Result<WebsiteResponse> {
let mut headers = HeaderMap::new();
headers.insert(
http::header::LOCATION,
HeaderValue::try_from(location)
.map_err(|_| S3Error::with_message(S3ErrorCode::InternalError, "invalid website redirect location"))?,
);
Ok(answer(status, headers, Body::empty()))
}
fn website_relative_redirect(uri: &Uri, path: &str, status: StatusCode) -> S3Result<WebsiteResponse> {
let mut url = url::Url::parse("http://localhost/").map_err(S3Error::internal_error)?;
url.set_path(path);
url.set_query(uri.query());
let location = match url.query() {
Some(query) => format!("{}?{query}", url.path()),
None => url.path().to_owned(),
};
website_location(location, status)
}
fn website_redirect(
uri: &Uri,
headers: &HeaderMap,
host: &str,
protocol: Option<&str>,
key: &str,
status: StatusCode,
website_scheme: &str,
) -> S3Result<WebsiteResponse> {
let scheme = protocol.unwrap_or(website_scheme);
let host = if host.is_empty() {
headers.get(HOST).and_then(|h| h.to_str().ok()).unwrap_or_default()
} else {
host
};
let mut url = url::Url::parse(&format!("{scheme}://{host}/")).map_err(S3Error::internal_error)?;
url.set_path(&format!("/{key}"));
url.set_query(uri.query());
website_location(url.into(), status)
}
fn website_rule_matches(rule: &RoutingRule, key: &str, status: StatusCode) -> bool {
let Some(condition) = rule.condition.as_ref() else { return true };
condition
.key_prefix_equals
.as_ref()
.is_none_or(|prefix| key.starts_with(prefix))
&& condition
.http_error_code_returned_equals
.as_ref()
.is_none_or(|code| code == status.as_str())
}
fn website_rule_redirect(req: &WebsiteRequest, rule: &RoutingRule, key: &str, website_scheme: &str) -> S3Result<WebsiteResponse> {
let redirect = &rule.redirect;
let key = if let Some(replace) = &redirect.replace_key_with {
replace.clone()
} else if let Some(replace) = &redirect.replace_key_prefix_with {
let prefix = rule
.condition
.as_ref()
.and_then(|condition| condition.key_prefix_equals.as_deref())
.unwrap_or_default();
format!("{replace}{}", key.strip_prefix(prefix).unwrap_or(key))
} else {
key.to_owned()
};
let status = redirect
.http_redirect_code
.as_deref()
.and_then(|code| code.parse::<u16>().ok())
.and_then(|code| StatusCode::from_u16(code).ok())
.unwrap_or(StatusCode::MOVED_PERMANENTLY);
website_redirect(
&req.uri,
&req.headers,
redirect.host_name.as_deref().unwrap_or_default(),
redirect.protocol.as_deref(),
&key,
status,
website_scheme,
)
}
#[cfg(test)]
mod tests;
+463
View File
@@ -0,0 +1,463 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! The website rules, and the host answering real buckets as both stacks serve them.
use super::*;
use crate::app::storage_api::bucket_usecase::contract::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::app::storage_api::object_usecase::contract::object::ObjectIO as _;
use crate::app::storage_api::object_usecase::{StorageObjectOptions, StoragePutObjReader};
use crate::app::storage_api::test::ECStore;
use http_body_util::BodyExt;
use rustfs_gateway_types::persistence::{
PersistedRedirect as Redirect, PersistedRoutingRule as RoutingRule, PersistedRoutingRuleCondition as Condition,
};
use serial_test::serial;
fn uri(value: &str) -> Uri {
value.parse().expect("test URI should parse")
}
fn host(value: &str) -> HeaderMap {
let mut headers = HeaderMap::new();
headers.insert(HOST, value.parse().expect("valid host"));
headers
}
#[test]
fn website_host_is_isolated_from_s3_host_and_keeps_directory_suffix() {
let domains = vec!["example.com".to_owned()];
let mut headers = host("share.example.com:9000");
let target = website_target(&uri("/docs/"), &headers, &domains).expect("website host matches");
assert_eq!(target.bucket, "share");
assert_eq!(target.key, "docs/");
assert!(target.trailing_slash);
headers.insert(HOST, "s3.other.test".parse().expect("valid host"));
assert!(website_target(&uri("/share/docs/"), &headers, &domains).is_none());
headers.insert(HOST, "my.bucket.example.com".parse().expect("valid host"));
assert_eq!(
website_target(&uri("/"), &headers, &domains)
.expect("dotted bucket host matches")
.bucket,
"my.bucket"
);
headers.insert(HOST, "example.com".parse().expect("valid host"));
assert_eq!(
website_target(&uri("/share/key"), &headers, &domains)
.expect("base domain is reserved")
.bucket,
""
);
headers.insert(HOST, "share.example.com".parse().expect("valid host"));
assert!(
website_target(&uri("/%FF"), &headers, &domains)
.expect("invalid path stays in website route")
.invalid_path
);
}
#[test]
fn website_host_claims_only_its_domains_and_nothing_without_one() {
let server_ctx = ServerContextSlot::new();
assert!(WebsiteHost::new(Vec::new(), "http", Arc::clone(&server_ctx)).is_none());
let website = WebsiteHost::new(vec!["website.test".to_owned()], "http", server_ctx).expect("a configured domain");
for claimed in [
"website.test",
"bucket.website.test",
"BUCKET.Website.Test.",
"bucket.website.test:9000",
] {
assert!(website.claims(&uri("/key"), &host(claimed)), "{claimed}");
}
for unclaimed in [
"localhost",
"website.test.evil",
"bucketwebsite.test",
"s3.example.test",
"a b",
] {
assert!(!website.claims(&uri("/key"), &host(unclaimed)), "{unclaimed}");
}
assert!(!website.claims(&uri("/key"), &HeaderMap::new()), "no Host");
}
#[test]
fn website_redirect_encodes_path_and_preserves_query() {
let target = uri("/old?download=1");
let headers = host("site.example.com");
let redirect = website_redirect(
&target,
&headers,
"new.example.com",
Some("https"),
"new path/#one",
StatusCode::MOVED_PERMANENTLY,
"http",
)
.expect("valid redirect");
assert_eq!(
redirect.headers()[http::header::LOCATION],
"https://new.example.com/new%20path/%23one?download=1"
);
let directory = website_relative_redirect(&target, "/new path/", StatusCode::FOUND).expect("valid directory redirect");
assert_eq!(directory.headers()[http::header::LOCATION], "/new%20path/?download=1");
let secure = website_redirect(&target, &headers, "new.example.com", None, "key", StatusCode::MOVED_PERMANENTLY, "https")
.expect("TLS listener default scheme");
assert_eq!(secure.headers()[http::header::LOCATION], "https://new.example.com/key?download=1");
let same_host = website_redirect(&target, &headers, "", None, "key", StatusCode::MOVED_PERMANENTLY, "http")
.expect("an empty host name keeps the request's host");
assert_eq!(same_host.headers()[http::header::LOCATION], "http://site.example.com/key?download=1");
}
#[test]
fn website_rules_choose_first_match_after_object_status_is_known() {
let error_rule = RoutingRule {
condition: Some(Condition {
http_error_code_returned_equals: Some("404".to_owned()),
key_prefix_equals: None,
}),
redirect: Redirect::default(),
};
let prefix_rule = RoutingRule {
condition: Some(Condition {
http_error_code_returned_equals: None,
key_prefix_equals: Some("docs/".to_owned()),
}),
redirect: Redirect::default(),
};
let rules = [error_rule, prefix_rule];
assert_eq!(
rules
.iter()
.position(|rule| website_rule_matches(rule, "docs/missing", StatusCode::NOT_FOUND)),
Some(0)
);
assert_eq!(
rules
.iter()
.position(|rule| website_rule_matches(rule, "docs/existing", StatusCode::OK)),
Some(1)
);
assert_eq!(
rules
.iter()
.position(|rule| website_rule_matches(rule, "other", StatusCode::FORBIDDEN)),
None
);
}
#[tokio::test]
async fn website_304_preserves_cache_headers_without_content() {
let mut error = S3Error::new(S3ErrorCode::NotModified);
let mut headers = HeaderMap::new();
for (name, value) in [
("etag", "\"abc\""),
("cache-control", "max-age=60"),
("expires", "Tue, 14 Nov 2023 22:13:20 GMT"),
("content-length", "123"),
("content-type", "application/xml"),
("transfer-encoding", "chunked"),
] {
headers.insert(name, HeaderValue::from_static(value));
}
error.set_headers(headers);
let response = website_not_modified(&error);
assert_eq!(response.status(), StatusCode::NOT_MODIFIED);
assert_eq!(response.headers()["etag"], "\"abc\"");
assert_eq!(response.headers()["cache-control"], "max-age=60");
assert!(response.headers().contains_key("expires"));
for dropped in ["content-length", "content-type", "transfer-encoding"] {
assert!(!response.headers().contains_key(dropped), "{dropped}");
}
assert!(
response
.into_body()
.collect()
.await
.expect("empty website body")
.to_bytes()
.is_empty()
);
}
#[test]
fn website_conditions_are_refused_as_the_s3_decoder_refuses_them() {
let refused = |name: &'static str, value: &'static str| {
let mut headers = HeaderMap::new();
headers.insert(name, HeaderValue::from_static(value));
validated_conditions(&headers).expect_err(value).code().clone()
};
assert_eq!(refused("range", "bytes=5-1"), S3ErrorCode::InvalidRange);
assert_eq!(refused("range", "bytes=0-1,3-4"), S3ErrorCode::InvalidRange);
assert_eq!(refused("range", "items=0-1"), S3ErrorCode::InvalidRange);
assert_eq!(refused("if-modified-since", "yesterday"), S3ErrorCode::InvalidArgument);
assert_eq!(refused("if-unmodified-since", "2024-01-01"), S3ErrorCode::InvalidArgument);
let mut headers = HeaderMap::new();
headers.insert("range", HeaderValue::from_static("bytes=2-"));
headers.insert("if-none-match", HeaderValue::from_static("*"));
headers.insert("if-modified-since", HeaderValue::from_static("Tue, 14 Nov 2023 22:13:20 GMT"));
assert!(matches!(
validated_conditions(&headers).expect("well-formed conditions"),
Some(Range::Int { first: 2, last: None })
));
let stripped = without_conditions(&headers);
assert!(stripped.is_empty(), "{stripped:?}");
}
/// A bucket served as a website, as `website_hosting_test` builds it.
struct Site {
website: WebsiteHost,
bucket: String,
}
const DOMAIN: &str = "website.test";
const PUBLIC_READ: &str = "public";
async fn put(store: &ECStore, bucket: &str, key: &str, body: &str, redirect: Option<&str>) {
let mut options = StorageObjectOptions::default();
if let Some(location) = redirect {
options
.user_defined
.insert(rustfs_utils::http::AMZ_WEBSITE_REDIRECT_LOCATION.to_owned(), location.to_owned());
}
store
.put_object(bucket, key, &mut StoragePutObjReader::from_vec(body.as_bytes().to_vec()), &options)
.await
.expect("seed a website object");
}
/// A fresh bucket holding the website pages, with `website` as its website configuration and,
/// when `policy` is `PUBLIC_READ`, the public-read policy that denies `private.txt`.
async fn site(website: Option<&str>, policy: &str) -> Site {
let store = crate::app::gating_test_env::shared_gating_ecstore().await;
let server_ctx = ServerContextSlot::new();
assert!(server_ctx.install(crate::app::gating_test_env::shared_gating_ambient().await));
let bucket = format!("website-{}", uuid::Uuid::new_v4());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create the website bucket");
for (key, body) in [
("index.html", "home"),
("404.html", "error page"),
("docs", "ordinary object"),
("section/index.html", "section page"),
("private.txt", "private"),
] {
put(store.as_ref(), &bucket, key, body, None).await;
}
put(store.as_ref(), &bucket, "move", "ordinary redirect object", Some("/new-page")).await;
if let Some(website) = website {
store
.update_bucket_metadata_config(
&bucket,
crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_WEBSITE_CONFIG,
website.as_bytes().to_vec(),
)
.await
.expect("store the website configuration");
}
if policy == PUBLIC_READ {
let policy = serde_json::json!({
"Version": "2012-10-17",
"Statement": [
{"Effect":"Allow","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/*")},
{"Effect":"Deny","Principal":"*","Action":"s3:GetObject","Resource":format!("arn:aws:s3:::{bucket}/private.txt")}
]
});
store
.update_bucket_metadata_config(
&bucket,
crate::storage::storage_api::ecstore_bucket::metadata::BUCKET_POLICY_CONFIG,
serde_json::to_vec(&policy).expect("policy JSON"),
)
.await
.expect("store the public-read policy");
}
let website = WebsiteHost::new(vec![DOMAIN.to_owned()], "http", server_ctx).expect("a website domain");
Site { website, bucket }
}
const INDEX_AND_ERROR: &str = "<WebsiteConfiguration><IndexDocument><Suffix>index.html</Suffix></IndexDocument><ErrorDocument><Key>404.html</Key></ErrorDocument></WebsiteConfiguration>";
/// Status, `Location`, and body of one website request.
struct Page {
status: StatusCode,
location: Option<String>,
body: String,
}
impl Site {
async fn ask(&self, method: Method, path: &str, extra: &[(&'static str, &'static str)]) -> Page {
self.ask_host(&format!("{}.{DOMAIN}", self.bucket), method, path, extra).await
}
async fn ask_host(&self, host_name: &str, method: Method, path: &str, extra: &[(&'static str, &'static str)]) -> Page {
let mut headers = host(host_name);
for (name, value) in extra {
headers.insert(*name, HeaderValue::from_static(value));
}
let response = self.website.answer(method, uri(path), headers, Extensions::new()).await;
let status = response.status();
let location = response
.headers()
.get(http::header::LOCATION)
.map(|value| value.to_str().expect("an ASCII location").to_owned());
let body = response.into_body().collect().await.expect("an in-memory body").to_bytes();
Page {
status,
location,
body: String::from_utf8_lossy(&body).into_owned(),
}
}
}
#[tokio::test]
#[serial]
async fn website_host_serves_pages_documents_and_redirects_anonymously() {
let site = site(Some(INDEX_AND_ERROR), PUBLIC_READ).await;
let home = site.ask(Method::GET, "/", &[]).await;
assert_eq!((home.status, home.body.as_str()), (StatusCode::OK, "home"));
let head = site.ask(Method::HEAD, "/", &[]).await;
assert_eq!((head.status, head.body.as_str()), (StatusCode::OK, ""));
let object = site.ask(Method::GET, "/docs", &[]).await;
assert_eq!((object.status, object.body.as_str()), (StatusCode::OK, "ordinary object"));
let section = site.ask(Method::GET, "/section/", &[]).await;
assert_eq!((section.status, section.body.as_str()), (StatusCode::OK, "section page"));
let missing = site.ask(Method::GET, "/missing", &[]).await;
assert_eq!((missing.status, missing.body.as_str()), (StatusCode::NOT_FOUND, "error page"));
let denied = site.ask(Method::GET, "/private.txt", &[]).await;
assert_eq!((denied.status, denied.body.as_str()), (StatusCode::FORBIDDEN, "error page"));
let head_missing = site.ask(Method::HEAD, "/missing", &[]).await;
assert_eq!((head_missing.status, head_missing.body.as_str()), (StatusCode::NOT_FOUND, ""));
let directory = site.ask(Method::GET, "/section?x=1", &[]).await;
assert_eq!(directory.status, StatusCode::FOUND);
assert_eq!(directory.location.as_deref(), Some("/section/?x=1"));
let redirect = site.ask(Method::GET, "/move", &[]).await;
assert_eq!(redirect.status, StatusCode::MOVED_PERMANENTLY);
assert_eq!(redirect.location.as_deref(), Some("/new-page"));
for condition in [("if-none-match", "*"), ("range", "bytes=999999-")] {
let redirected = site.ask(Method::GET, "/move", &[condition]).await;
assert_eq!(redirected.status, StatusCode::MOVED_PERMANENTLY, "{condition:?}");
assert_eq!(redirected.location.as_deref(), Some("/new-page"), "{condition:?}");
}
let unsatisfiable = site.ask(Method::GET, "/index.html", &[("range", "bytes=999999-")]).await;
// A client error is answered by the error document, under the error's own status.
assert_eq!(
(unsatisfiable.status, unsatisfiable.body.as_str()),
(StatusCode::RANGE_NOT_SATISFIABLE, "error page")
);
let ranged = site.ask(Method::GET, "/index.html", &[("range", "bytes=1-2")]).await;
assert_eq!((ranged.status, ranged.body.as_str()), (StatusCode::PARTIAL_CONTENT, "om"));
let malformed = site
.ask(Method::GET, "/index.html", &[("if-modified-since", "yesterday")])
.await;
assert_eq!(malformed.status, StatusCode::BAD_REQUEST);
assert!(!malformed.body.contains("<Error>"), "{}", malformed.body);
let post = site.ask(Method::POST, "/", &[]).await;
assert_eq!(post.status, StatusCode::METHOD_NOT_ALLOWED);
let base_domain = site
.ask_host(DOMAIN, Method::GET, &format!("/{}/index.html", site.bucket), &[])
.await;
assert_eq!(base_domain.status, StatusCode::NOT_FOUND);
assert!(!base_domain.body.contains("<Error>"), "{}", base_domain.body);
let no_bucket = site
.ask_host(&format!("absent-{}.{DOMAIN}", site.bucket), Method::GET, "/", &[])
.await;
assert_eq!(no_bucket.status, StatusCode::NOT_FOUND);
}
#[tokio::test]
#[serial]
async fn website_host_refuses_what_the_bucket_does_not_serve() {
// No website configuration: the bucket is not a website, whatever its policy.
let unconfigured = site(None, PUBLIC_READ).await;
let page = unconfigured.ask(Method::GET, "/", &[]).await;
assert_eq!(page.status, StatusCode::NOT_FOUND);
assert_eq!(page.body, "<html><body>Not Found</body></html>");
// A website without a public-read policy is read as the anonymous caller it is: the index and
// the error document are both refused, so the page is the plain refusal.
let private = site(Some(INDEX_AND_ERROR), "none").await;
for path in ["/", "/index.html", "/docs", "/missing"] {
let page = private.ask(Method::GET, path, &[]).await;
assert_eq!(page.status, StatusCode::FORBIDDEN, "{path}");
assert_eq!(page.body, "<html><body>Forbidden</body></html>", "{path}");
}
// Whatever caller and ownership the request arrived with, a website read is anonymous.
let mut owner = Extensions::new();
owner.insert(ReqInfo {
cred: Some(rustfs_credentials::Credentials {
access_key: "website-owner".to_owned(),
..Default::default()
}),
is_owner: true,
..ReqInfo::default()
});
let host_name = format!("{}.{DOMAIN}", private.bucket);
let response = private.website.answer(Method::GET, uri("/"), host(&host_name), owner).await;
assert_eq!(
response.status(),
StatusCode::FORBIDDEN,
"an owner's request facts must not reach the read"
);
// A redirect location is only reported to a read the bucket allows.
let hidden = private.ask(Method::GET, "/move", &[]).await;
assert_eq!((hidden.status, hidden.location), (StatusCode::FORBIDDEN, None));
// An index document that does not exist is answered by the error document's status.
let no_index = site(
Some("<WebsiteConfiguration><IndexDocument><Suffix>absent.html</Suffix></IndexDocument></WebsiteConfiguration>"),
PUBLIC_READ,
)
.await;
let page = no_index.ask(Method::GET, "/", &[]).await;
assert_eq!(page.status, StatusCode::NOT_FOUND);
assert_eq!(page.body, "<html><body>Not Found</body></html>");
}
#[tokio::test]
#[serial]
async fn website_host_applies_routing_rules_and_redirect_all() {
let rules = "<WebsiteConfiguration><IndexDocument><Suffix>index.html</Suffix></IndexDocument><RoutingRules>\
<RoutingRule><Condition><HttpErrorCodeReturnedEquals>404</HttpErrorCodeReturnedEquals></Condition><Redirect><ReplaceKeyWith>missing-page</ReplaceKeyWith></Redirect></RoutingRule>\
<RoutingRule><Condition><KeyPrefixEquals>docs/</KeyPrefixEquals></Condition><Redirect><ReplaceKeyWith>docs-page</ReplaceKeyWith></Redirect></RoutingRule>\
<RoutingRule><Condition><HttpErrorCodeReturnedEquals>416</HttpErrorCodeReturnedEquals></Condition><Redirect><ReplaceKeyWith>range-error</ReplaceKeyWith></Redirect></RoutingRule>\
</RoutingRules></WebsiteConfiguration>";
let site = site(Some(rules), PUBLIC_READ).await;
let host_name = format!("{}.{DOMAIN}", site.bucket);
let rule = site.ask(Method::GET, "/docs/missing", &[]).await;
assert_eq!(rule.status, StatusCode::MOVED_PERMANENTLY);
assert_eq!(rule.location, Some(format!("http://{host_name}/missing-page")));
let ranged = site.ask(Method::GET, "/index.html", &[("range", "bytes=999999-")]).await;
assert_eq!(ranged.status, StatusCode::MOVED_PERMANENTLY);
assert_eq!(ranged.location, Some(format!("http://{host_name}/range-error")));
let served = site.ask(Method::GET, "/index.html", &[]).await;
assert_eq!((served.status, served.body.as_str()), (StatusCode::OK, "home"));
let all = site_redirecting_all().await;
let page = all.ask(Method::GET, "/a%20b?x=1", &[]).await;
assert_eq!(page.status, StatusCode::MOVED_PERMANENTLY);
assert_eq!(page.location.as_deref(), Some("http://destination.test/a%20b?x=1"));
}
async fn site_redirecting_all() -> Site {
site(
Some("<WebsiteConfiguration><RedirectAllRequestsTo><HostName>destination.test</HostName></RedirectAllRequestsTo></WebsiteConfiguration>"),
"none",
)
.await
}
+1
View File
@@ -162,6 +162,7 @@ impl IAMAuth {
}
}
#[cfg(test)]
pub(crate) fn with_server_context(
ak: impl Into<String>,
sk: impl Into<SecretKey>,
+21 -62
View File
@@ -25,70 +25,35 @@ use rustfs_config::{
use rustfs_credentials::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, Masked};
use rustfs_policy::auth::{is_access_key_valid, is_secret_key_valid};
use std::collections::HashSet;
use std::fmt;
use std::io::{Error, ErrorKind};
use std::str::FromStr;
use std::sync::{Mutex, OnceLock};
pub(crate) const LEGACY_ENV_RUSTFS_ROOT_USER: &str = "RUSTFS_ROOT_USER";
pub(crate) const LEGACY_ENV_RUSTFS_ROOT_PASSWORD: &str = "RUSTFS_ROOT_PASSWORD";
static LEGACY_CREDENTIAL_WARNED_KEYS: OnceLock<Mutex<HashSet<String>>> = OnceLock::new();
/// Development-only selector between the s3s-based S3 stack and the gateway
/// stack while both coexist on the integration branch (rustfs/backlog#2734).
/// It is removed when the gateway stack becomes the only one.
const ENV_RUSTFS_S3_STACK: &str = "RUSTFS_S3_STACK";
/// The selector between the s3s-based S3 stack and the gateway stack while both
/// existed (rustfs/backlog#2734). The gateway stack is the only one now, so a set
/// selector, whatever its value, refuses startup rather than being ignored.
pub const RETIRED_S3_STACK_SELECTOR: &str = "RUSTFS_S3_STACK";
/// The S3 protocol stack that serves requests.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
pub enum S3Stack {
/// The s3s-based stack every release ships today.
#[default]
Legacy,
/// The rustfs-gateway stack under construction.
Gateway,
}
/// A `RUSTFS_S3_STACK` value other than the exact names `legacy` and `gateway`.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct InvalidS3Stack {
value: String,
}
impl fmt::Display for InvalidS3Stack {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
write!(f, r#"{ENV_RUSTFS_S3_STACK} must be "legacy" or "gateway", got {:?}"#, self.value)
}
}
impl std::error::Error for InvalidS3Stack {}
impl FromStr for S3Stack {
type Err = InvalidS3Stack;
fn from_str(value: &str) -> Result<Self, Self::Err> {
// Exact names only: a near miss must refuse startup rather than fall
// back to a stack the operator did not ask for.
match value {
"legacy" => Ok(Self::Legacy),
"gateway" => Ok(Self::Gateway),
_ => Err(InvalidS3Stack { value: value.to_owned() }),
}
}
}
impl S3Stack {
/// Unset selects the default; a set but invalid value, including an empty
/// one, is a startup error.
pub(crate) fn from_env() -> std::io::Result<Self> {
match std::env::var_os(ENV_RUSTFS_S3_STACK) {
None => Ok(Self::default()),
Some(value) => value
.to_string_lossy()
.parse()
.map_err(|err| std::io::Error::new(std::io::ErrorKind::InvalidInput, err)),
}
/// Refuses startup when the retired S3 stack selector is set to any value,
/// the empty string included.
///
/// # Errors
///
/// `InvalidInput` naming the variable and this release when it is set.
pub(crate) fn refuse_retired_s3_stack_selector() -> std::io::Result<()> {
if std::env::var_os(RETIRED_S3_STACK_SELECTOR).is_none() {
return Ok(());
}
Err(Error::new(
ErrorKind::InvalidInput,
format!(
"{RETIRED_S3_STACK_SELECTOR} is no longer supported; the gateway stack is the only S3 stack since {}",
env!("CARGO_PKG_VERSION")
),
))
}
fn warn_legacy_credential_env_once(legacy_key: &str, canonical_key: &str) {
@@ -209,9 +174,6 @@ pub struct Config {
/// Workload profile for adaptive buffer sizing
pub buffer_profile: String,
/// S3 protocol stack selected by `RUSTFS_S3_STACK`
pub s3_stack: S3Stack,
}
impl Config {
@@ -243,7 +205,6 @@ impl Config {
kms_allow_insecure_dev_defaults: false,
buffer_profile_disable: false,
buffer_profile: "GeneralPurpose".to_string(),
s3_stack: S3Stack::default(),
}
}
@@ -308,7 +269,7 @@ impl Config {
// Region is optional, but if not set, we should default to "us-east-1" for signing compatibility with AWS S3 clients
let region = region.or_else(|| Some(RUSTFS_REGION.to_string()));
let s3_stack = S3Stack::from_env()?;
refuse_retired_s3_stack_selector()?;
Ok(Config {
volumes,
@@ -333,7 +294,6 @@ impl Config {
kms_allow_insecure_dev_defaults,
buffer_profile_disable,
buffer_profile,
s3_stack,
})
}
@@ -377,7 +337,6 @@ impl std::fmt::Debug for Config {
.field("kms_allow_insecure_dev_defaults", &self.kms_allow_insecure_dev_defaults)
.field("buffer_profile_disable", &self.buffer_profile_disable)
.field("buffer_profile", &self.buffer_profile)
.field("s3_stack", &self.s3_stack)
.finish()
}
}

Some files were not shown because too many files have changed in this diff Show More