gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-10-27 14:27:10 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge pinned main into gateway integration

Browse Source 
Merge main acd26afd8b into
db0828aa1c. Keep the gateway runtime and
recovery assertions, restore the original OIDC assertions in the IAM and
admin tests, and refresh both guardrail baselines from measured output.

The three former OIDC service-only states are no longer representable
after the paired service/query change. Their removal is not counted as
equivalent execution coverage.

All 20 original local source-gate commands completed with child exit 0:
- shellcheck --version && actionlint (exit 0)
- cargo fmt --all --check (exit 0)
- ./scripts/check_unsafe_code_allowances.sh (exit 0)
- ./scripts/check_layer_dependencies.sh (exit 0)
- ./scripts/check_architecture_migration_rules.sh (exit 0)
- ./scripts/check_logging_guardrails.sh (exit 0)
- ./scripts/check_error_other_format_ratchet.sh (exit 0)
- ./scripts/check_no_tokio_io_uring.sh (exit 0)
- ./scripts/check_extension_schema_boundaries.sh (exit 0)
- ./scripts/check_body_cache_whitelist.sh (exit 0)
- ./scripts/check_s3s_footprint.sh (exit 0)
- ./scripts/check_test_count_ratchet.sh (exit 0)
- ./scripts/check_fips_wording.sh (exit 0)
- ./scripts/check_embedded_secrets.sh (exit 0)
- make script-tests (exit 0)
- python3 ./scripts/test_object_generation_protocol_model.py (exit 0)
- python3 ./scripts/check_test_wiring.py (exit 0)
- ./scripts/check_no_planning_docs.sh (exit 0)
- ./scripts/check_uring_lane_lib_only.sh (exit 0)
- python3 ./scripts/ci_gate.py --check-workflow (exit 0)

Earlier runs stopped for a formatting mismatch, the measured S3S footprint
baseline mismatch, and capacity limits, including an interrupted script-test
run. Those original results remain recorded; this was not a first-pass run.
Full remote branch CI has not run yet.
This commit is contained in:
overtrue
2026-10-12 10:19:25 +08:00
parent db0828aa1c acd26afd8b
commit daef49b42e
119 changed files with 11009 additions and 7432 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/security-advisory-lessons/references/advisory-patterns.md
+6 -3
View File
@@ -45,11 +45,13 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
### S3 object actions, copy, multipart, and upload policy validation
- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -110,6 +112,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
### Serde deserialization and input validation
@@ -143,13 +146,13 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions.
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.
.config/e2e-nightly-selection.txt
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-darwin=efe68675f4676a0d2901242c5815b1f7af5cdec8231f7cccd408f6dacf5c75db
sha256-linux=003ed6480935a0a95680cc061be143722fbd37bd477b8ff98607b1a064f3aef1
sha256-darwin=5dbb9154b8feb94cedc8fb4aa6895d4b3259cc31b7df1f5c3b9928ef07f04535
sha256-linux=d3a210be9eccf593d3723ae669b66680e9751dd01a6949f03df20614f913b707
.config/make/tests.mak
+1
View File
@@ -54,6 +54,7 @@ script-tests: ## Run shell script tests
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_migration_gate_evidence.py
$(RUSTFS_PYTHON_BIN) ./scripts/ci_gate.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_security_coverage.py --self-test
.config/nextest.toml
+10
View File
@@ -92,6 +92,12 @@ setup = 'ecstore-large-stack'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
# These handler/store fixtures and the relocated-pool GET compose deep storage
# futures in libtest threads; on Linux they overflow the default 2 MiB stack.
[[profile.default.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.default.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
@@ -337,6 +343,10 @@ setup = 'ecstore-large-stack'
filter = 'package(rustfs-ecstore) | package(rustfs-s3select-api) | package(rustfs-scanner) | (package(rustfs) & test(/^(app::multipart_usecase::tests::concurrent_completions_share_durable_bucket_quota_reservations|app::object::delete::tests::compressed_delete_requests_update_observed_usage_without_releasing_quota_floor|app::object::internal_put::tests::internal_multipart_roundtrip_completes_and_abort_leaves_nothing|app::object::restore::tests::execute_restore_object_maps_failures_to_typed_s3_errors|storage::access::tests::(delete_object_access_captures_authorized_bucket_incarnation|copy_operations_reject_recreated_source_bucket_after_authorization|request_slot_keeps_bucket_policy_bound_to_its_store))$/))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'package(rustfs) & (test(/^admin::handlers::table_catalog::tests::/) | test(/^table_catalog::store::tests::/) | test(=app::object::get::tests::execute_get_object_resumes_from_relocated_pool_without_splicing_body))'
setup = 'ecstore-base-stack'
[[profile.ci.scripts]]
filter = 'binary(lifecycle_integration_test) | (package(rustfs) & test(/^app::lifecycle_transition_api_test::/))'
setup = 'lifecycle-large-stack'
.config/s3s-footprint-baseline.txt
+1 -1
View File
@@ -13,5 +13,5 @@
# header's rule clears (rustfs_s3_types's macro); fails closed
# ecstore_files Rust files referencing s3s paths under crates/ecstore/src
files=194
s3_error_lines=1584
s3_error_lines=1566
ecstore_files=36
.config/test-count-baseline.txt
+11 -11
View File
@@ -9,8 +9,8 @@
# the PR that adds tests; lowering one is a reviewed statement that tests were
# intentionally removed. Regenerate with:
# scripts/check_test_count_ratchet.sh --update
e2e_test 890 18
rustfs 5540 62
e2e_test 892 18
rustfs 5536 62
rustfs-audit 70 0
rustfs-checksums 26 0
rustfs-common 23 0
@@ -20,12 +20,12 @@ rustfs-connect-symbol-catalog 6 0
rustfs-credentials 24 0
rustfs-crypto 67 0
rustfs-data-usage 54 0
rustfs-ecstore 6307 1
rustfs-ecstore 6322 1
rustfs-extension-schema 23 0
rustfs-filemeta 306 0
rustfs-heal 839 3
rustfs-heal-contracts 6 0
rustfs-iam 371 0
rustfs-iam 377 0
rustfs-io-core 49 0
rustfs-io-metrics 125 0
rustfs-keystone 29 0
@@ -33,25 +33,25 @@ rustfs-kms 715 25
rustfs-license 2 0
rustfs-lifecycle 152 0
rustfs-lock 135 0
rustfs-log-analyzer 87 0
rustfs-log-analyzer 88 0
rustfs-madmin 209 0
rustfs-notify 136 0
rustfs-object-capacity 92 0
rustfs-object-data-cache 140 0
rustfs-obs 365 0
rustfs-policy 677 0
rustfs-obs 366 0
rustfs-policy 680 0
rustfs-protocols 736 1
rustfs-protos 52 0
rustfs-replication 200 0
rustfs-rio 198 0
rustfs-rio 200 0
rustfs-rio-v2 34 6
rustfs-s3-client 65 0
rustfs-s3-ops 8 0
rustfs-s3-types 97 0
rustfs-s3select-api 206 0
rustfs-s3select-query 188 2
rustfs-scanner 971 20
rustfs-scanner-metrics 75 0
rustfs-scanner 991 20
rustfs-scanner-metrics 76 0
rustfs-security-governance 20 0
rustfs-signer 31 0
rustfs-storage-api 31 0
@@ -59,5 +59,5 @@ rustfs-targets 603 26
rustfs-test-utils 1 0
rustfs-tls-runtime 20 0
rustfs-trusted-proxies 68 0
rustfs-utils 216 0
rustfs-utils 217 0
rustfs-zip 30 0
.github/workflows/build.yml
+11
View File
@@ -440,6 +440,17 @@ jobs:
fi
done
- name: Verify macOS binary has no dynamic liblzma dependency
if: matrix.platform == 'macos'
shell: bash
run: |
binary="target/${{ matrix.target }}/release/rustfs"
if otool -L "$binary" | grep -E 'liblzma.*\.dylib'; then
echo "macOS release binary must not depend on a dynamic liblzma library" >&2
exit 1
fi
"$binary" --version
- name: Generate the CPU symbol catalog
if: endsWith(matrix.target, '-linux-gnu') && matrix.features == 'pyroscope'
shell: bash
.github/workflows/cache-warm.yml
+1 -1
View File
@@ -99,7 +99,7 @@ env:
RUST_BACKTRACE: 1
jobs:
# Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
# Readers: test-and-lint (workspace and ILM), build-rustfs-debug-binary,
# e2e-tests, e2e-full.
warm-ci-dev:
name: Warm ci-dev
.github/workflows/ci.yml
+50 -84
View File
@@ -170,13 +170,10 @@ jobs:
echo "started_at=$(date --utc --iso-8601=seconds)"
} > artifacts/test-and-lint/run-metadata.txt
# Clippy runs before the test pass: lint failures are the most common
# CI-only breakage and should surface in minutes, not after 20+ minutes
# of tests.
# Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
# experiment, since --all-targets is check-only for workspace members and
# never links the ~100 test binaries the limit exists to throttle.
# Keep full Clippy validation on main, merge groups, and scheduled/manual
# runs. PRs retain the complete default test suite and compile-free checks.
- name: Run clippy lints
if: github.event_name != 'pull_request'
run: |
./scripts/ci/resource_sampler.sh start clippy
trap './scripts/ci/resource_sampler.sh stop' EXIT
@@ -285,6 +282,49 @@ jobs:
- name: Verify rebalance/decommission migration proofs
run: ./scripts/check_migration_gate_count.sh evidence artifacts/test-and-lint/core-test-listing.json target/nextest/ci/junit.xml
# The workspace already compiled these ignored tests. Keep the same Cargo
# package selection to reuse their binaries, then run them serially in
# separate processes because they share fixed ports and global state.
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
if: needs.classify-changes.outputs.mode == 'full'
env:
# Match the preceding workspace build and reuse its feature graph.
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run --profile default -j1 --run-ignored ignored-only \
--all --exclude e2e_test \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always() && needs.classify-changes.outputs.mode == 'full'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: artifacts/ilm-integration
retention-days: 3
# This gate builds into a fresh target directory to isolate the E2E root.
# Give its cold build a separate budget from workspace tests and migration proofs.
offline-enrollment-e2e:
@@ -316,83 +356,11 @@ jobs:
- name: Check offline enrollment E2E root boundary
run: ./scripts/check_offline_enrollment_e2e.sh
# Dedicated serial lane for the ILM / lifecycle integration tests. These tests
# drive the object layer through process-global singletons (the GLOBAL_ENV
# ECStore, the global tier-config manager, background-expiry workers) and bind
# fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
# marked #[ignore] to keep them out of the default parallel `cargo nextest run
# --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
# `-j1` so no two of them share global state or race for a port at once.
# serial_test's #[serial] does NOT serialize across nextest's process-per-test
# boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
# See rustfs/backlog#1148 (ilm-1) and #1155.
test-ilm-integration-serial:
name: ILM Integration (serial)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
env:
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: ci-dev
cache-save-if: 'false'
install-build-packaging-tools: 'false'
# test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
# its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
# disk_index 0), not an EC metadata-distribution issue.
# restore_object_usecase_reports_ongoing_conflict_and_completion was
# re-enabled by backlog#1304 (restore accepts serialize on a short CAS
# guard; the copy-back no longer holds the #4877 whole-copy-back lock,
# so the mid-restore ongoing read and fast 409 rejection it asserts are
# the implemented contract).
- name: Run ignored ILM integration tests serially
env:
# Match the measured Test and Lint link budget. The default exposed
# all 14 pod CPUs and a cold cache spent the full 80m compiling
# without starting one ILM test (main run 32982910990).
CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
run: |
mkdir -p artifacts/ilm-integration
set +e
NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 80m \
cargo nextest run -j1 --run-ignored ignored-only \
-p rustfs-scanner -p rustfs \
-E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))' \
--status-level all --final-status-level all \
2>&1 | tee artifacts/ilm-integration/nextest.log
status=${PIPESTATUS[0]}
{
echo "exit_status=${status}"
echo "finished_at=$(date --utc --iso-8601=seconds)"
echo
echo "Remaining test-related processes:"
pgrep -af 'cargo|nextest|target/.*/deps/' || true
} > artifacts/ilm-integration/diagnostics.txt
exit "${status}"
- name: Upload ILM test diagnostics
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ilm-integration-${{ github.run_number }}-${{ github.run_attempt }}
path: |
artifacts/ilm-integration
target/nextest/ci/junit.xml
test-and-lint-rio-v2:
name: Test and Lint (rio-v2)
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
# Alternate feature configurations run on main, merge groups, and scheduled
# or manual validation; the default workspace remains the PR gate.
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -508,7 +476,7 @@ jobs:
test-and-lint-protocols:
name: "Test and Lint (${{ matrix.features.name }})"
if: needs.classify-changes.outputs.mode == 'full' && (github.event_name != 'pull_request' || github.event.action != 'closed')
if: needs.classify-changes.outputs.mode == 'full' && github.event_name != 'pull_request'
needs: [ quick-checks, classify-changes ]
runs-on: sm-standard-4
timeout-minutes: 90
@@ -1355,7 +1323,6 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- connect-short-credential-boundary
- test-and-lint-protocols
@@ -1389,7 +1356,6 @@ jobs:
- quick-checks
- test-and-lint
- offline-enrollment-e2e
- test-ilm-integration-serial
- test-and-lint-rio-v2
- test-and-lint-protocols
- build-rustfs-debug-binary
.github/workflows/e2e-upgrade.yml
+37 -41
View File
@@ -96,38 +96,8 @@ jobs:
overwrite: true
upgrade:
name: ${{ matrix.name }}
name: Upgrade compatibility
needs: build
strategy:
fail-fast: false
matrix:
include:
# The two `_from_rc2_` tests keep their names: they assert
# release-independent object contracts and pass unchanged against the
# newer pinned source, so renaming them would only churn history and
# the CI required-check names. UPGRADE_SOURCE_VERSION above is the
# single source of truth for which release they actually run against.
- name: Direct upgrade from the previous release
test: direct_upgrade_from_rc2_preserves_object_contracts
artifact: direct-upgrade
- name: Mixed-version rolling upgrade from the previous release
test: rolling_upgrade_from_rc2_preserves_mixed_version_contracts
artifact: mixed-version-upgrade
- name: Bucket configuration survives the upgrade
test: direct_upgrade_from_previous_release_preserves_bucket_configuration
artifact: bucket-config-upgrade
- name: Rollback reads current bucket metadata
test: rollback_to_previous_release_reads_current_bucket_metadata
artifact: bucket-config-rollback
- name: ODM configuration recovery after rc.5 rollback
test: rc5_rollback_requires_restoring_odm_configuration
artifact: odm-config-rollback
- name: Multipart layouts survive the rc.5 upgrade
test: direct_upgrade_from_rc5_preserves_multipart_layouts
artifact: multipart-layout-upgrade
- name: rc.5 multipart replication baseline
test: rc5_baseline_replicates_multipart_layouts
artifact: multipart-layout-baseline
runs-on: sm-standard-2
timeout-minutes: 60
env:
@@ -142,8 +112,7 @@ jobs:
uses: ./.github/actions/setup
with:
cache-shared-key: e2e-upgrade-tests
# One main-branch consumer warms the shared test-harness dependencies.
cache-save-if: ${{ github.ref == 'refs/heads/main' && matrix.artifact == 'direct-upgrade' }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
install-build-packaging-tools: "false"
install-test-tools: "false"
@@ -170,22 +139,49 @@ jobs:
echo "RUSTFS_UPGRADE_SOURCE_BINARY=$SOURCE_DIR/rustfs" >> "$GITHUB_ENV"
echo "RUSTFS_E2E_LOG_DIR=$RUNNER_TEMP/rustfs-upgrade-logs" >> "$GITHUB_ENV"
- name: Run upgrade compatibility test
# Reuse one compiled harness, while keeping each case in its own process.
# Continue after a failure to retain the former matrix's complete verdict.
- name: Run upgrade compatibility tests
shell: bash
env:
RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
NO_PROXY: 127.0.0.1,localhost,::1
run: |
export no_proxy="$NO_PROXY"
python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::${{ matrix.test }}" \
-- --ignored --exact --nocapture
cases=(
direct-upgrade:direct_upgrade_from_rc2_preserves_object_contracts
mixed-version-upgrade:rolling_upgrade_from_rc2_preserves_mixed_version_contracts
bucket-config-upgrade:direct_upgrade_from_previous_release_preserves_bucket_configuration
bucket-config-rollback:rollback_to_previous_release_reads_current_bucket_metadata
odm-config-rollback:rc5_rollback_requires_restoring_odm_configuration
multipart-layout-upgrade:direct_upgrade_from_rc5_preserves_multipart_layouts
multipart-layout-baseline:rc5_baseline_replicates_multipart_layouts
)
status=0
for entry in "${cases[@]}"; do
artifact="${entry%%:*}"
test="${entry#*:}"
export RUSTFS_SCANNER_HEAL_G09_EVIDENCE_DIR="$RUNNER_TEMP/rustfs-upgrade-g09-evidence/$artifact"
export RUSTFS_E2E_LOG_DIR="$RUNNER_TEMP/rustfs-upgrade-logs/$artifact"
mkdir -p "$RUSTFS_E2E_LOG_DIR"
echo "::group::$test"
if python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test \
"upgrade_compatibility_test::$test" -- --ignored --exact --nocapture \
2>&1 | tee "$RUSTFS_E2E_LOG_DIR/test.log"; then
echo "PASS: $test"
else
echo "::error::Upgrade compatibility failed: $test"
status=1
fi
echo "::endgroup::"
done
exit "$status"
- name: Upload scanner/heal G09 evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence/${{ matrix.artifact }}
name: upgrade-g09-evidence-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-g09-evidence
if-no-files-found: ignore
retention-days: 14
@@ -193,7 +189,7 @@ jobs:
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: ${{ matrix.artifact }}-server-logs-${{ github.run_number }}
name: upgrade-server-logs-${{ github.run_number }}
path: ${{ runner.temp }}/rustfs-upgrade-logs
if-no-files-found: warn
retention-days: 14
.github/workflows/oidc-keycloak.yml
+2 -2
View File
@@ -20,7 +20,7 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
@@ -31,7 +31,7 @@ on:
- ".github/workflows/oidc-keycloak.yml"
- "crates/config/src/constants/oidc.rs"
- "crates/iam/src/federation/**"
- "crates/iam/src/oidc.rs"
- "crates/iam/src/oidc/**"
- "rustfs/src/admin/handlers/oidc.rs"
- "rustfs/src/admin/handlers/sts.rs"
- "scripts/test/oidc_keycloak_live.sh"
.vscode/launch.json
Vendored
+21 -27
View File
@@ -7,7 +7,9 @@
"name": "Debug(only) executable 'rustfs'",
"env": {
"RUST_LOG": "rustfs=info,ecstore=info,s3s=info,iam=info",
"RUSTFS_SKIP_BACKGROUND_TASK": "on"
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
// Local test volumes share one physical disk.
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true"
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -44,6 +46,7 @@
"env": {
"RUST_LOG": "rustfs=debug,ecstore=info,s3s=debug,iam=debug",
"RUSTFS_SKIP_BACKGROUND_TASK": "on",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
//"RUSTFS_OBS_LOG_DIRECTORY": "./deploy/logs",
// "RUSTFS_POLICY_PLUGIN_URL":"http://localhost:8181/v1/data/rustfs/authz/allow",
// "RUSTFS_POLICY_PLUGIN_AUTH_TOKEN":"your-opa-token"
@@ -117,7 +120,8 @@
"env": {
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
//"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_VOLUMES": "./target/volume/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
// "RUSTFS_OBS_TRACE_ENDPOINT": "http://127.0.0.1:4318/v1/traces", // jeager otlp http endpoint
@@ -135,41 +139,30 @@
{
"type": "lldb",
"request": "launch",
"name": "Debug test_lifecycle_transition_basic",
"name": "Debug test_transition_and_restore_flows",
"cargo": {
"args": [
"test",
"--no-run",
"-p",
"rustfs-scanner",
"--test",
"lifecycle_integration_test",
"serial_tests::test_lifecycle_transition_basic",
"-j",
"1"
]
],
"filter": {
"name": "lifecycle_integration_test",
"kind": "test"
}
},
"args": [],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/test",
"type": "lldb",
"request": "launch",
"program": "${workspaceFolder}/target/debug/deps/lifecycle_integration_test-5915cbfcab491b3b",
"args": [
"--skip",
"test_lifecycle_expiry_basic",
"--skip",
"test_lifecycle_expiry_deletemarker",
//"--skip",
//"test_lifecycle_transition_basic",
],
"cwd": "${workspaceFolder}",
//"stopAtEntry": false,
//"preLaunchTask": "cargo build",
"sourceLanguages": [
"rust"
"serial_tests::test_transition_and_restore_flows",
"--exact",
"--ignored",
"--test-threads=1"
],
"cwd": "${workspaceFolder}"
},
{
"name": "Debug executable target/debug/rustfs with sse kms",
@@ -285,7 +278,7 @@
"RUSTFS_COMPRESSION_ENABLED": "true",
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/rio-v2/kms-key-dir",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/rio-v2/kms-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
@@ -307,6 +300,7 @@
"RUSTFS_ACCESS_KEY": "rustfsadmin",
"RUSTFS_SECRET_KEY": "rustfsadmin",
"RUSTFS_VOLUMES": "./target/e2e-test/test{1...4}",
"RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
"RUSTFS_REGION": "us-east-1",
"RUSTFS_ADDRESS": ":9000",
"RUSTFS_CONSOLE_ENABLE": "true",
@@ -315,7 +309,7 @@
"RUSTFS_KMS_ENABLE": "true",
"RUSTFS_KMS_BACKEND": "local",
"RUSTFS_KMS_KEY_DIR": "./target/e2e-key-dir",
"RUSTFS_KMS_KEY_DIR": "${workspaceFolder}/target/e2e-key-dir",
"RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
"RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
},
Cargo.lock
Generated
+3
View File
@@ -10045,6 +10045,7 @@ dependencies = [
"tokio",
"tokio-util",
"tracing",
"tracing-subscriber",
"url",
]
@@ -10595,6 +10596,7 @@ dependencies = [
"faster-hex",
"futures",
"hex-simd",
"hmac 0.13.0",
"hotpath",
"http 1.5.0",
"http-body-util",
@@ -11073,6 +11075,7 @@ dependencies = [
"async-compression",
"futures",
"hotpath",
"liblzma",
"rustfs-rio",
"rustfs-tokio-tar",
"serde",
crates/e2e_test/src/common.rs
+9
View File
@@ -1542,6 +1542,15 @@ impl RustFSTestClusterEnvironment {
self.start_node_from_binary(node_idx, &binary_path).await
}
/// Start a process and wait only for TCP, allowing tests to bring up a
/// quorum one node at a time before the S3 service can become ready.
pub async fn start_node_process(&mut self, node_idx: usize) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
let binary_path = rustfs_binary_path();
let volumes_arg = self.build_volumes_arg();
self.spawn_node(node_idx, &binary_path, &volumes_arg)?;
self.wait_for_node_ready(&self.nodes[node_idx].address, node_idx).await
}
/// Start one stopped cluster node with a specific RustFS binary while
/// preserving the cluster's volume layout and that node's data directory.
pub async fn start_node_from_binary(
crates/e2e_test/src/namespace_lock_quorum_test.rs
+281 -3
View File
@@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::common::RustFSTestClusterEnvironment;
use crate::common::{ClusterTopology, RustFSTestClusterEnvironment};
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use bytes::Bytes;
@@ -185,7 +185,11 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
return Ok(());
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("read quorum did not become available after lease convergence for {bucket}").into());
let diagnostics = clients[0].get_object().bucket(bucket).key("warm-small").send().await;
return Err(format!(
"read quorum did not become available after lease convergence for {bucket}: GET={diagnostics:?}"
)
.into());
}
tokio::time::sleep(std::time::Duration::from_millis(100)).await;
}
@@ -195,11 +199,14 @@ async fn wait_for_quorum_read_admission(clients: &[Client], bucket: &str) -> Tes
async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(4, 2), (6, 3), (6, 2)] {
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
// Wait for every seed fanout before removing any physical shard.
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
@@ -338,6 +345,277 @@ async fn test_degraded_cluster_read_quorum_follows_erasure_layout() -> TestResul
Ok(())
}
#[tokio::test]
async fn test_cold_start_read_quorum_follows_erasure_layout() -> TestResult {
crate::common::init_logging();
for (node_count, parity) in [(3, 1), (4, 2), (6, 3), (6, 2), (4, 0)] {
let read_quorum = node_count - parity;
let write_quorum = read_quorum + usize::from(read_quorum == parity);
let mut cluster = RustFSTestClusterEnvironment::new(node_count).await?;
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", format!("EC:{parity}"));
if parity == 0 {
cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
}
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "warn");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = format!("cold-read-quorum-{node_count}-{parity}");
clients[0].create_bucket().bucket(&bucket).send().await?;
let small = b"cold startup follows the stored EC layout".to_vec();
let large = vec![73; 256 * 1024];
for (key, body) in [
("warm-small", &small),
("warm-large", &large),
("cold-large", &large),
("below-quorum", &large),
] {
clients[0]
.put_object()
.bucket(&bucket)
.key(key)
.body(Bytes::copy_from_slice(body).into())
.send()
.await?;
for node in &cluster.nodes {
let census =
crate::chaos::census_object_version_on_disk(std::path::Path::new(&node.data_dir), &bucket, key, None)?;
assert!(census.is_complete(), "seed shard must exist before cold startup: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(read_quorum), Some(parity)));
}
}
for node in 0..node_count {
cluster.stop_node(node)?;
}
// Rebuild every process's in-memory state below read quorum, then
// cross the EC boundary. A format bootstrap must not admit S3 reads
// merely because half the peers are online (e.g. six disks, EC:2).
for node in 0..read_quorum - 1 {
cluster.start_node_process(node).await?;
}
let error = clients[0]
.get_object()
.bucket(&bucket)
.key("below-quorum")
.send()
.await
.expect_err("cold startup below the configured EC read quorum must stay unavailable");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(read_quorum - 1).await?;
wait_for_quorum_read_admission(&clients[..read_quorum], &bucket).await?;
for client in clients.iter().take(read_quorum) {
assert_quorum_object_body(client, &bucket, "warm-small", &small).await?;
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
let write = client
.put_object()
.bucket(&bucket)
.key("cold-quorum-write")
.body(Bytes::copy_from_slice(&small).into())
.send()
.await;
if read_quorum >= write_quorum {
write?;
assert_quorum_object_body(client, &bucket, "cold-quorum-write", &small).await?;
} else {
let error = write.expect_err("cold startup at read quorum must preserve the EC write quorum");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
}
for node in read_quorum..node_count {
cluster.start_node(node).await?;
}
for client in &clients {
assert_quorum_object_body(client, &bucket, "cold-large", &large).await?;
}
}
Ok(())
}
#[tokio::test]
async fn test_four_by_six_cold_start_at_read_quorum() -> TestResult {
crate::common::init_logging();
let mut cluster = RustFSTestClusterEnvironment::with_topology(ClusterTopology::single_pool_multidrive(4, 6)).await?;
// Explicit URLs are consumed in order. Place one drive from each node in
// every set, matching a four-host, six-drive-per-host deployment.
let volumes = (0..6)
.flat_map(|drive| {
cluster
.nodes
.iter()
.map(move |node| format!("http://{}{}", node.address, node.data_dirs[drive]))
})
.collect::<Vec<_>>()
.join(" ");
cluster.set_env("RUSTFS_VOLUMES", volumes);
cluster.set_env("RUSTFS_ERASURE_SET_DRIVE_COUNT", "4");
cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
cluster.set_env("RUSTFS_PUT_RENAME_EARLY_ACK_ENABLE", "false");
cluster.set_env("RUSTFS_OBS_METRICS_EXPORT_ENABLED", "false");
cluster.set_env("RUST_LOG", "error");
cluster.start().await?;
let clients = cluster
.create_all_clients()?
.into_iter()
.map(|client| {
Client::from_conf(
client
.config()
.to_builder()
.retry_config(aws_sdk_s3::config::retry::RetryConfig::standard().with_max_attempts(1))
.build(),
)
})
.collect::<Vec<_>>();
let bucket = "cold-start-read-quorum";
clients[0].create_bucket().bucket(bucket).send().await?;
let mut fixtures = Vec::new();
let mut covered = [[false; 2]; 6];
for index in 0..128 {
let key = match index {
0 => "warm-small".to_string(),
1 => "warm-large".to_string(),
_ => format!("cold-{index}"),
};
let size_class = index % 2;
let size = if size_class == 0 { 71 } else { 256 * 1024 };
let body = (0..size)
.map(|offset| u8::try_from(offset % 251).expect("bounded fixture byte"))
.collect::<Vec<_>>();
clients[0]
.put_object()
.bucket(bucket)
.key(&key)
.body(Bytes::copy_from_slice(&body).into())
.send()
.await?;
for node in &cluster.nodes {
let mut complete_shards = 0;
for (drive, path) in node.data_dirs.iter().enumerate() {
let census = crate::chaos::census_object_version_on_disk(std::path::Path::new(path), bucket, &key, None)?;
if census.has_xl_meta {
assert!(census.is_complete(), "seed shard must be complete before shutdown: {census:?}");
assert_eq!((census.data_blocks, census.parity_blocks), (Some(2), Some(2)));
covered[drive][size_class] = true;
complete_shards += 1;
}
}
assert_eq!(complete_shards, 1, "each node must own one shard of every seed object");
}
fixtures.push((key, body));
if covered.iter().flatten().all(|covered| *covered) {
break;
}
}
assert!(
covered.iter().flatten().all(|covered| *covered),
"fixtures must cover all six sets with inline and large objects"
);
for first in 0..4 {
for second in first + 1..4 {
for node in 0..4 {
cluster.stop_node(node)?;
}
cluster.start_node_process(first).await?;
let error = clients[first]
.get_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("one node cannot satisfy read quorum after a cold start");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
cluster.start_node_process(second).await?;
wait_for_quorum_read_admission(&[clients[first].clone(), clients[second].clone()], bucket).await?;
for node in [first, second] {
for (key, body) in &fixtures {
assert_quorum_object_body(&clients[node], bucket, key, body).await?;
}
let range = clients[node]
.get_object()
.bucket(bucket)
.key("warm-large")
.range("bytes=17-1023")
.send()
.await?;
assert_eq!(range.body.collect().await?.into_bytes().as_ref(), &fixtures[1].1[17..1024]);
let listing = clients[node].list_objects_v2().bucket(bucket).send().await?;
for (key, _) in &fixtures {
assert!(
listing.contents().iter().any(|object| object.key() == Some(key.as_str())),
"listing omitted {key}"
);
}
let error = clients[node]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
.expect_err("two nodes must not authorize a write");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
}
let error = clients[first]
.delete_object()
.bucket(bucket)
.key("warm-large")
.send()
.await
.expect_err("two nodes must not authorize a delete");
assert_eq!(error.raw_response().map(|response| response.status().as_u16()), Some(503));
assert_quorum_object_body(&clients[first], bucket, "warm-large", &fixtures[1].1).await?;
let remaining = (0..4).filter(|node| *node != first && *node != second).collect::<Vec<_>>();
cluster.start_node(remaining[0]).await?;
let deadline = tokio::time::Instant::now() + Duration::from_secs(90);
loop {
match clients[first]
.put_object()
.bucket(bucket)
.key("recovery-write")
.body(Bytes::from_static(b"after recovery").into())
.send()
.await
{
Ok(_) => break,
Err(error)
if error.raw_response().is_some_and(|response| response.status().as_u16() == 503)
&& tokio::time::Instant::now() < deadline =>
{
tokio::time::sleep(Duration::from_millis(200)).await
}
Err(error) => return Err(error.into()),
}
}
for node in [first, second, remaining[0]] {
assert_quorum_object_body(&clients[node], bucket, "recovery-write", b"after recovery").await?;
}
cluster.start_node(remaining[1]).await?;
for client in &clients {
for (key, body) in &fixtures {
assert_quorum_object_body(client, bucket, key, body).await?;
}
assert_quorum_object_body(client, bucket, "recovery-write", b"after recovery").await?;
}
}
}
Ok(())
}
async fn put_object(client: Client, payload: Vec<u8>, writer_id: usize) -> Result<(), String> {
client
.put_object()
crates/e2e_test/src/sts_query_compat_test.rs
+1 -1
View File
@@ -740,7 +740,7 @@ async fn test_bucket_tags_opa_contract() -> TestResult {
.version_id(version)
.send()
.await?;
opa.expect_bucket_tags("s3:GetObject", source, Some("finance")).await?;
opa.expect_bucket_tags("s3:GetObjectVersion", source, Some("finance")).await?;
client.list_objects_v2().bucket(source).send().await?;
opa.expect_bucket_tags("s3:ListBucket", source, Some("finance")).await?;
client.list_object_versions().bucket(source).send().await?;
crates/ecstore/src/api/mod.rs
+2 -1
View File
@@ -210,7 +210,8 @@ pub mod bucket {
pub mod migration {
pub use crate::bucket::migration::{
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_iam_config,
LegacyBlobDecryptFn, migration_startup_error, try_migrate_bucket_metadata, try_migrate_bucket_metadata_for_buckets,
try_migrate_iam_config,
};
}
crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs
+29 -6
View File
@@ -2939,6 +2939,20 @@ fn resolve_tier_free_version_recovery_enabled(value: Result<String, env::VarErro
}
fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<()>) -> Option<JoinHandle<()>> {
spawn_tier_free_version_recovery_once_with_options(
api,
started,
TIER_FREE_VERSION_RECOVERY_LEADER_LOCK.to_string(),
tier_free_version_recovery_leader_retry_delay,
)
}
fn spawn_tier_free_version_recovery_once_with_options(
api: Arc<ECStore>,
started: &OnceLock<()>,
leader_lock_name: String,
retry_delay: fn() -> StdDuration,
) -> Option<JoinHandle<()>> {
if !tier_free_version_recovery_enabled() {
warn!(
event = EVENT_LIFECYCLE_WORKER_STATE,
@@ -2956,15 +2970,15 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>, started: &OnceLock<(
Some(tokio::spawn(async move {
let cancel_token = api.ctx.background_cancel_token().unwrap_or_default();
let acquire_api = Arc::clone(&api);
let acquire_lock_name = Arc::<str>::from(leader_lock_name);
run_tier_free_version_recovery_coordinator(
cancel_token,
tier_free_version_recovery_leader_retry_delay,
retry_delay,
move || {
let api = Arc::clone(&acquire_api);
let lock_name = Arc::clone(&acquire_lock_name);
async move {
let lock = api
.new_ns_lock(RUSTFS_META_BUCKET, TIER_FREE_VERSION_RECOVERY_LEADER_LOCK)
.await?;
let lock = api.new_ns_lock(RUSTFS_META_BUCKET, &lock_name).await?;
match lock.get_write_lock_quiet(TIER_FREE_VERSION_RECOVERY_LOCK_TIMEOUT).await {
Ok(guard) => Ok(Some(guard)),
Err(LockError::Timeout { .. }) => Ok(None),
@@ -14193,6 +14207,10 @@ mod tests {
.await;
}
fn fast_tier_free_version_recovery_leader_retry_delay() -> StdDuration {
StdDuration::from_millis(10)
}
#[tokio::test]
#[serial]
async fn tier_free_version_recovery_production_entrypoint_enqueues_seeded_item() {
@@ -14212,8 +14230,13 @@ mod tests {
seed_recoverable_free_version(&disk_paths, &bucket, object, None, None).await;
let started = OnceLock::new();
let recovery = super::spawn_tier_free_version_recovery_once(Arc::clone(&ecstore), &started)
.expect("production recovery entrypoint should start once");
let recovery = super::spawn_tier_free_version_recovery_once_with_options(
Arc::clone(&ecstore),
&started,
format!("tier-free-version-recovery/test/{}", Uuid::new_v4()),
fast_tier_free_version_recovery_leader_retry_delay,
)
.expect("production recovery entrypoint should start once");
let task = tokio::time::timeout(StdDuration::from_secs(30), async { recovery_rx.lock().await.recv().await })
.await
.expect("production recovery entrypoint should enqueue the seeded item")
crates/ecstore/src/bucket/metadata_sys.rs
+24 -4
View File
@@ -1292,14 +1292,34 @@ pub(crate) async fn get_object_lock_config_state_in(
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
get_object_lock_config_and_incarnation_from_disk_with_guard_in(ctx, bucket, None).await
}
/// Bucket deletion already owns the namespace/lifecycle fences. Reuse its
/// live guard rather than recursively acquiring the bucket namespace lock.
/// The caller must also hold the metadata transaction read lock.
pub(crate) async fn get_object_lock_config_and_incarnation_from_disk_with_guard_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
guard: Option<&rustfs_lock::NamespaceLockGuard>,
) -> Result<(ObjectLockConfigState, Uuid, OffsetDateTime)> {
let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
let bucket_meta_sys = bucket_meta_sys_lock.read().await.clone();
match bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
{
let authority = match guard {
Some(guard) => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_guard(bucket, guard)
.await?
}
None => {
bucket_meta_sys
.read_authoritative_metadata_from_disk_under_transaction_lock(bucket)
.await?
}
};
match authority {
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() =>
{
crates/ecstore/src/bucket/migration.rs
+24
View File
@@ -273,6 +273,30 @@ where
let buckets: Vec<String> = buckets_list.into_iter().map(|b| b.name).collect();
try_migrate_bucket_metadata_for_buckets(store, &buckets).await
}
/// Migrate the already discovered startup candidates without repeating a
/// majority-only namespace listing. Every migration write retains its quorum.
pub async fn try_migrate_bucket_metadata_for_buckets<S>(store: Arc<S>, buckets: &[String]) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
> + ObjectOperations<
Error = Error,
ObjectInfo = ObjectInfo,
ObjectOptions = ObjectOptions,
FileInfo = FileInfo,
ObjectToDelete = ObjectToDelete,
DeletedObject = DeletedObject,
>,
{
if buckets.is_empty() {
debug!("No migrating bucket metadata found");
return Ok(());
crates/ecstore/src/disk/error_reduce.rs
+27 -4
View File
@@ -131,6 +131,15 @@ pub fn build_write_quorum_failure_summary(
fn dominant_error_label(errors: &[Option<Error>], ignored_errs: &[Error], dominant_error: Option<&Error>) -> &'static str {
let Some(dominant_error) = dominant_error else {
// The reducer also returns None when every failure is ignored for
// quorum voting. That case contains no successful nil entry.
if !errors.is_empty() && errors.iter().all(Option::is_some) {
return if errors.iter().all(|error| error.as_ref() == Some(&Error::DiskNotFound)) {
"disk_not_found"
} else {
"all_ignored"
};
}
return "nil_dominated";
};
@@ -387,9 +396,9 @@ mod tests {
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_is_nil_dominated() {
// All disks offline (all DiskNotFound). The dominant error must be None and
// the label must be "nil_dominated" rather than falling back to "other_error".
fn test_build_write_quorum_failure_summary_all_missing_is_disk_not_found() {
// Ignoring unavailable disks for quorum reduction must not imply a
// successful nil entry in the diagnostic label.
let errors = vec![
Some(Error::DiskNotFound),
Some(Error::DiskNotFound),
@@ -397,7 +406,21 @@ mod tests {
];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.dominant_error_label, "nil_dominated");
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "disk_not_found");
}
#[test]
fn test_build_write_quorum_failure_summary_all_ignored_has_no_success() {
let errors = [Some(Error::FaultyDisk), Some(Error::DiskNotFound)];
let summary = build_write_quorum_failure_summary(&errors, OBJECT_OP_IGNORED_ERRS, 2);
assert_eq!(summary.dominant_error, None);
assert_eq!(summary.achieved, 0);
assert_eq!(summary.dominant_error_label, "all_ignored");
assert_eq!(
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
Some(Error::ErasureWriteQuorum)
);
}
#[test]
crates/ecstore/src/io_support/rio.rs
+53 -10
View File
@@ -442,12 +442,19 @@ impl WritePlan {
}
if let Some(encryption) = self.encryption {
let etag_nonce = match encryption.mode {
WriteEncryptionMode::Singlepart { base_nonce } | WriteEncryptionMode::MultipartLegacy { base_nonce, .. } => {
base_nonce
}
WriteEncryptionMode::SinglepartObjectKey | WriteEncryptionMode::MultipartObjectKey { .. } => [0u8; 12],
};
let protected_reader = rustfs_rio::EtagReader::new_encrypted(reader, encryption.key_bytes, etag_nonce)?;
reader = match encryption.mode {
WriteEncryptionMode::SinglepartObjectKey => HashReader::from_reader(
#[cfg(feature = "rio-v2")]
EncryptReader::new_with_object_key(reader, encryption.key_bytes),
EncryptReader::new_with_object_key(protected_reader, encryption.key_bytes),
#[cfg(not(feature = "rio-v2"))]
EncryptReader::new(reader, encryption.key_bytes, [0u8; 12]),
EncryptReader::new(protected_reader, encryption.key_bytes, [0u8; 12]),
HashReader::SIZE_PRESERVE_LAYER,
actual_size,
None,
@@ -457,12 +464,12 @@ impl WritePlan {
WriteEncryptionMode::Singlepart { base_nonce } => {
#[cfg(not(feature = "rio-v2"))]
let encrypt_reader = if encryption_frame_v2_enabled() {
EncryptReader::new_v2(reader, encryption.key_bytes, base_nonce)
EncryptReader::new_v2(protected_reader, encryption.key_bytes, base_nonce)
} else {
EncryptReader::new(reader, encryption.key_bytes, base_nonce)
EncryptReader::new(protected_reader, encryption.key_bytes, base_nonce)
};
#[cfg(feature = "rio-v2")]
let encrypt_reader = EncryptReader::new(reader, encryption.key_bytes, base_nonce);
let encrypt_reader = EncryptReader::new(protected_reader, encryption.key_bytes, base_nonce);
HashReader::from_reader(encrypt_reader, HashReader::SIZE_PRESERVE_LAYER, actual_size, None, None, false)?
}
WriteEncryptionMode::MultipartLegacy {
@@ -471,20 +478,25 @@ impl WritePlan {
} => {
#[cfg(not(feature = "rio-v2"))]
let encrypt_reader = if encryption_frame_v2_enabled() {
EncryptReader::new_multipart_v2(reader, encryption.key_bytes, base_nonce, multipart_part_number)
EncryptReader::new_multipart_v2(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number)
} else {
EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number)
EncryptReader::new_multipart(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number)
};
#[cfg(feature = "rio-v2")]
let encrypt_reader =
EncryptReader::new_multipart(reader, encryption.key_bytes, base_nonce, multipart_part_number);
EncryptReader::new_multipart(protected_reader, encryption.key_bytes, base_nonce, multipart_part_number);
HashReader::from_reader(encrypt_reader, HashReader::SIZE_PRESERVE_LAYER, actual_size, None, None, false)?
}
WriteEncryptionMode::MultipartObjectKey { multipart_part_number } => HashReader::from_reader(
#[cfg(feature = "rio-v2")]
EncryptReader::new_multipart_with_object_key(reader, encryption.key_bytes, multipart_part_number),
EncryptReader::new_multipart_with_object_key(protected_reader, encryption.key_bytes, multipart_part_number),
#[cfg(not(feature = "rio-v2"))]
EncryptReader::new_multipart(reader, encryption.key_bytes, [0u8; 12], multipart_part_number as usize),
EncryptReader::new_multipart(
protected_reader,
encryption.key_bytes,
[0u8; 12],
multipart_part_number as usize,
),
HashReader::SIZE_PRESERVE_LAYER,
actual_size,
None,
@@ -497,6 +509,7 @@ impl WritePlan {
// `ignore_value` deliberately avoids a second hasher over compressed or
// encrypted bytes. The inner reader still validates the plaintext request
// checksum while this outer reader exposes the request checksum context.
reader.encrypted_etag = encrypted;
reader.add_non_trailing_checksum(checksum, true)?;
reader.set_trailer(trailer);
@@ -513,6 +526,36 @@ mod tests {
use std::io::Cursor;
use tokio::io::AsyncReadExt;
#[tokio::test]
async fn write_plan_encryption_persists_protected_etags_for_every_mode() {
let data = b"Salary 2026: 85000 EUR";
let size = i64::try_from(data.len()).expect("test data size");
let mut control =
HashReader::from_stream(Cursor::new(data), size, size, None, None, false).expect("create control reader");
control.read_to_end(&mut Vec::new()).await.expect("read control plaintext");
let md5 = control.try_resolve_etag().expect("control MD5");
for encryption in [
WriteEncryption::singlepart([1; 32], [1; 12]),
WriteEncryption::singlepart_object_key([1; 32]),
WriteEncryption::multipart([1; 32], [1; 12], 1),
WriteEncryption::multipart_object_key([1; 32], 1),
] {
let plaintext = HashReader::from_stream(Cursor::new(data), size, size, Some(md5.clone()), None, false)
.expect("create checked reader");
let mut reader = WritePlan::new()
.with_encryption(encryption)
.apply(plaintext, size)
.expect("apply encryption");
let mut ciphertext = Vec::new();
reader.read_to_end(&mut ciphertext).await.expect("stream ciphertext");
assert_ne!(ciphertext, data);
let tag = reader.try_resolve_etag().expect("resolve persisted encrypted ETag");
assert_ne!(tag, md5, "stored ETag must not reveal the plaintext MD5");
assert_eq!(tag.len(), 32, "multipart ETags require a 16-byte hexadecimal part tag");
assert_eq!(reader.try_resolve_etag().as_ref(), Some(&tag));
}
}
async fn assert_non_trailing_checksum_survives(plan: WritePlan) {
let plaintext = b"checksum-context-through-write-plan".repeat(256);
let actual_size = plaintext.len() as i64;
crates/ecstore/src/set_disk/ops/bucket.rs
+16 -2
View File
@@ -123,8 +123,18 @@ impl SetDisks {
}
pub(crate) async fn list_bucket_for_scanner(&self, _opts: &BucketOptions) -> Result<(Vec<BucketInfo>, bool)> {
self.list_bucket_with_quorum(BucketInfoQuorum::Write).await
}
pub(crate) async fn list_bucket_with_quorum(&self, quorum: BucketInfoQuorum) -> Result<(Vec<BucketInfo>, bool)> {
let disks = self.disk_inventory().await;
let write_quorum = (disks.len() / 2) + 1;
// Discovery uses the same namespace read boundary as stat_bucket.
// It grants no authority to mutate buckets or accept object reads.
let required = match quorum {
BucketInfoQuorum::Read => disks.len().div_ceil(2).max(1),
BucketInfoQuorum::Write => write_quorum,
};
let mut futures = Vec::with_capacity(disks.len());
for disk in disks {
@@ -153,7 +163,11 @@ impl SetDisks {
}
}
if let Some(err) = reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, write_quorum) {
let error = match quorum {
BucketInfoQuorum::Read => reduce_read_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
BucketInfoQuorum::Write => reduce_write_quorum_errs(&errs, BUCKET_OP_IGNORED_ERRS, required),
};
if let Some(err) = error {
return Err(err.into());
}
@@ -179,7 +193,7 @@ impl SetDisks {
topology_complete &= counts.values().all(|(count, _)| *count >= write_quorum);
let mut buckets = counts
.into_values()
.filter_map(|(count, bucket)| (count >= write_quorum).then_some(bucket))
.filter_map(|(count, bucket)| (count >= required).then_some(bucket))
.collect::<Vec<_>>();
buckets.sort_by(|left, right| left.name.cmp(&right.name));
Ok((buckets, topology_complete))
crates/ecstore/src/set_disk/ops/heal.rs
+489 -304
View File
@@ -99,6 +99,7 @@ static HEAL_RENAME_FAILURES: std::sync::Mutex<Vec<(String, String, usize)>> = st
#[derive(Clone, Copy, PartialEq, Eq)]
enum ReadRepairPausePhase {
Data,
PartStaging(usize),
Commit,
}
@@ -250,7 +251,16 @@ fn should_fail_heal_rename(_bucket: &str, _object: &str, _disk_index: usize) ->
}
#[cfg(test)]
static HEAL_WRITER_FAILURES: std::sync::Mutex<Vec<(String, String, usize, DiskError)>> = std::sync::Mutex::new(Vec::new());
struct HealWriterFailure {
bucket: String,
object: String,
part_number: Option<usize>,
disk_index: usize,
error: DiskError,
}
#[cfg(test)]
static HEAL_WRITER_FAILURES: std::sync::Mutex<Vec<HealWriterFailure>> = std::sync::Mutex::new(Vec::new());
#[cfg(test)]
struct HealWriterFailureScope {
@@ -260,21 +270,23 @@ struct HealWriterFailureScope {
#[cfg(test)]
impl HealWriterFailureScope {
fn install(bucket: &str, object: &str, disk_indexes: &[usize], error: DiskError) -> Self {
fn install(bucket: &str, object: &str, part_number: Option<usize>, disk_indexes: &[usize], error: DiskError) -> Self {
let mut failures = HEAL_WRITER_FAILURES
.lock()
.expect("heal writer failure registry should not poison");
assert!(
!failures.iter().any(|(registered_bucket, registered_object, _, _)| {
registered_bucket == bucket && registered_object == object
}),
!failures
.iter()
.any(|failure| failure.bucket == bucket && failure.object == object),
"heal writer failures must be installed once per object"
);
failures.extend(
disk_indexes
.iter()
.map(|index| (bucket.to_string(), object.to_string(), *index, error.clone())),
);
failures.extend(disk_indexes.iter().map(|index| HealWriterFailure {
bucket: bucket.to_string(),
object: object.to_string(),
part_number,
disk_index: *index,
error: error.clone(),
}));
Self {
bucket: bucket.to_string(),
object: object.to_string(),
@@ -288,25 +300,28 @@ impl Drop for HealWriterFailureScope {
HEAL_WRITER_FAILURES
.lock()
.expect("heal writer failure registry should not poison")
.retain(|(bucket, object, _, _)| bucket != &self.bucket || object != &self.object);
.retain(|failure| failure.bucket != self.bucket || failure.object != self.object);
}
}
#[cfg(test)]
fn injected_heal_writer_error(bucket: &str, object: &str, disk_index: usize) -> Option<DiskError> {
fn injected_heal_writer_error(bucket: &str, object: &str, part_number: usize, disk_index: usize) -> Option<DiskError> {
let mut failures = HEAL_WRITER_FAILURES
.lock()
.expect("heal writer failure registry should not poison");
failures
.iter()
.position(|(registered_bucket, registered_object, registered_index, _)| {
registered_bucket == bucket && registered_object == object && *registered_index == disk_index
.position(|failure| {
failure.bucket == bucket
&& failure.object == object
&& failure.disk_index == disk_index
&& failure.part_number.is_none_or(|part| part == part_number)
})
.map(|position| failures.swap_remove(position).3)
.map(|position| failures.swap_remove(position).error)
}
#[cfg(not(test))]
fn injected_heal_writer_error(_bucket: &str, _object: &str, _disk_index: usize) -> Option<DiskError> {
fn injected_heal_writer_error(_bucket: &str, _object: &str, _part_number: usize, _disk_index: usize) -> Option<DiskError> {
None
}
@@ -518,30 +533,64 @@ fn is_retryable_heal_writer_error(error: &DiskError) -> bool {
) || error.is_retryable_internode_write_failure()
}
fn warn_heal_writer_failures(
bucket: &str,
object: &str,
version_id: &str,
writer_failure_count: usize,
result: &'static str,
first_failure: &(usize, usize, String),
) {
let (first_part_number, first_disk_index, first_error) = first_failure;
warn!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket,
object,
version_id,
writer_failure_count,
first_part_number,
first_disk_index,
error = %first_error,
result,
state = "writer_unavailable",
"Set disk object heal writer failures"
);
struct HealWriterFailures {
target_count: usize,
count: usize,
first: Option<(usize, usize, String)>,
error: Option<DiskError>,
}
impl HealWriterFailures {
fn new(target_count: usize) -> Self {
Self {
target_count,
count: 0,
first: None,
error: None,
}
}
fn record(&mut self, part_number: usize, disk_index: usize, error: DiskError) {
self.count += 1;
if self.first.is_none() {
self.first = Some((part_number, disk_index, heal_writer_error_summary(&error)));
}
// A permanent failure must not be hidden behind an earlier transient
// target failure when the remaining targets also become unavailable.
if self
.error
.as_ref()
.is_none_or(|first| is_retryable_heal_writer_error(first) && !is_retryable_heal_writer_error(&error))
{
self.error = Some(error);
}
}
fn into_error(self) -> DiskError {
self.error.unwrap_or(DiskError::ErasureWriteQuorum)
}
fn warn(&self, bucket: &str, object: &str, version_id: &str, result: &'static str) {
let Some((first_part_number, first_disk_index, first_error)) = &self.first else {
return;
};
warn!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
state = "writer_unavailable",
result,
bucket,
object,
version_id,
target_count = self.target_count,
writer_failure_count = self.count,
first_part_number,
first_disk_index,
error = %first_error,
"Set disk object heal writer failures"
);
}
}
impl SetDisks {
@@ -1548,196 +1597,169 @@ impl SetDisks {
let dst_data_dir = data_dir;
if !latest_meta.deleted && !latest_meta.is_remote() {
let erasure_info = latest_meta.erasure.clone();
let mut writer_failure_count = 0usize;
let mut first_writer_failure = None;
let mut first_writer_error = None;
let mut all_writer_failures_retryable = true;
let mut writer_failure_warned = false;
let stage_result: disk::error::Result<Option<DiskError>> = async {
let erasure_info = latest_meta.erasure.clone();
let mut writer_failures = HealWriterFailures::new(disks_to_heal_count);
for (part_index, part) in latest_meta.parts.iter().enumerate() {
use crate::io_support::shard_integrity::{PartProofReader, PreparedIntegrity, ShardVerifier};
let proof = part
.integrity
.as_ref()
.map(|part| {
PartProofReader::new(part.clone(), &parts_metadata, &latest_disks, bucket, object)
})
.transpose()
.map_err(DiskError::from)?;
// This reader covers the whole encoded shard. A range
// offset can exclude the final encoded byte when the
// logical part ends inside a data block, allowing a
// one-byte tail truncation to evade deep Heal.
let till_offset = usize::try_from(erasure.shard_file_size(part.size as i64))
.map_err(|_| DiskError::FileCorrupt)?;
let use_mmap_read = object_mmap_read_enabled();
for (part_index, part) in latest_meta.parts.iter().enumerate() {
use crate::io_support::shard_integrity::{PartProofReader, PreparedIntegrity, ShardVerifier};
let proof = part
.integrity
.as_ref()
.map(|part| {
PartProofReader::new(part.clone(), &parts_metadata, &latest_disks, bucket, object)
})
.transpose()?;
// Cover the complete encoded shard, including a logical
// part's partial final data block, when checking truncation.
let till_offset = usize::try_from(erasure.shard_file_size(part.size as i64))
.map_err(|_| DiskError::FileCorrupt)?;
let use_mmap_read = object_mmap_read_enabled();
let mut readers = Vec::with_capacity(latest_disks.len());
let mut writers = Vec::with_capacity(out_dated_disks.len());
// let mut errors = Vec::with_capacity(out_dated_disks.len());
let mut writers = Vec::with_capacity(out_dated_disks.len());
let mut prefer = vec![false; latest_disks.len()];
for (index, disk) in latest_disks.iter().enumerate() {
// Preserve the committed layout: recomputing inline-ness here
// (with a hardcoded unversioned threshold) makes healed replicas
// diverge from healthy ones in quorum identity, so heal would
// flag them forever.
let is_inline_buffer = latest_meta.inline_data();
// create writers for all disk positions, but only for outdated disks
for (index, disk_op) in out_dated_disks.iter().enumerate() {
if let Some(outdated_disk) = disk_op {
let writer_result = if let Some(error) =
injected_heal_writer_error(bucket, object, part.number, index)
{
Err(error)
} else {
create_bitrot_writer(
is_inline_buffer,
Some(outdated_disk),
RUSTFS_META_TMP_BUCKET,
&path_join_buf(&[
&tmp_id.to_string(),
&dst_data_dir.to_string(),
&format!("part.{}", part.number),
]),
erasure.shard_file_size(part.size as i64),
erasure.shard_size(),
HashAlgorithm::HighwayHash256S,
)
.await
};
let writer = match writer_result {
Ok(writer) => writer,
Err(err) => {
writer_failures.record(part.number, index, err);
writers.push(None);
continue;
}
};
let mut writer = writer;
if let Some(proof) = &proof {
let verifier = ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)?;
writer.set_integrity(verifier);
}
writers.push(Some(writer));
} else {
writers.push(None);
}
}
// Heal each part. erasure.Heal() will write the healed
// part to .rustfs/tmp/uuid/ which needs to be renamed
// later to the final location.
if writers.iter().all(Option::is_none) {
writer_failures.warn(bucket, object, version_id, "all_targets_unavailable");
// No target can accept this part. Preserve the setup
// cause before opening source readers or reconstructing.
return Ok(Some(writer_failures.into_error()));
}
#[cfg(test)]
pause_read_repair(bucket, object, ReadRepairPausePhase::PartStaging(part.number)).await;
let mut readers = Vec::with_capacity(latest_disks.len());
let mut prefer = vec![false; latest_disks.len()];
let this_part_errs =
Self::shuffle_check_parts(&data_errs_by_part[&part_index], &erasure_info.distribution);
if this_part_errs[index] != CHECK_PART_SUCCESS {
trace!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
part_number = part.number,
disk_index = index,
part_status = this_part_errs[index],
state = "source_shard_skipped",
"Set disk source shard skipped"
);
readers.push(None);
continue;
}
if let (Some(disk), Some(metadata)) = (disk, &copy_parts_metadata[index]) {
let checksum_info = metadata.erasure.get_checksum_info(part.number);
let checksum_algo = if metadata.uses_legacy_checksum
&& checksum_info.algorithm == HashAlgorithm::HighwayHash256S
{
HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
match create_bitrot_reader(
metadata.data.as_deref(),
Some(disk),
bucket,
&path_join_buf(&[object, &src_data_dir, &format!("part.{}", part.number)]),
0,
till_offset,
erasure.shard_size(),
checksum_algo.clone(),
false,
use_mmap_read,
)
.await
{
Ok(Some(mut reader)) => {
if let Some(proof) = &proof {
reader
.set_integrity(
ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)
.map_err(DiskError::from)?,
)
.map_err(DiskError::from)?;
}
readers.push(Some(reader));
}
Ok(None) => {
readers.push(None);
continue;
}
Err(_e) => {
readers.push(None);
continue;
}
for (index, disk) in latest_disks.iter().enumerate() {
if this_part_errs[index] != CHECK_PART_SUCCESS {
trace!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
part_number = part.number,
disk_index = index,
part_status = this_part_errs[index],
state = "source_shard_skipped",
"Set disk source shard skipped"
);
readers.push(None);
continue;
}
prefer[index] = disk.host_name().is_empty();
} else {
readers.push(None);
// errors.push(Some(DiskError::DiskNotFound));
}
}
if let (Some(disk), Some(metadata)) = (disk, &copy_parts_metadata[index]) {
let checksum_info = metadata.erasure.get_checksum_info(part.number);
let checksum_algo = if metadata.uses_legacy_checksum
&& checksum_info.algorithm == HashAlgorithm::HighwayHash256S
{
HashAlgorithm::HighwayHash256SLegacy
} else {
checksum_info.algorithm
};
// Preserve the committed layout: recomputing inline-ness here
// (with a hardcoded unversioned threshold) makes healed replicas
// diverge from healthy ones in quorum identity, so heal would
// flag them forever.
let is_inline_buffer = latest_meta.inline_data();
// create writers for all disk positions, but only for outdated disks
for (index, disk_op) in out_dated_disks.iter().enumerate() {
if let Some(outdated_disk) = disk_op {
let writer_result = if let Some(error) = injected_heal_writer_error(bucket, object, index)
{
Err(error)
} else {
create_bitrot_writer(
is_inline_buffer,
Some(outdated_disk),
RUSTFS_META_TMP_BUCKET,
&path_join_buf(&[
&tmp_id.to_string(),
&dst_data_dir.to_string(),
&format!("part.{}", part.number),
]),
erasure.shard_file_size(part.size as i64),
match create_bitrot_reader(
metadata.data.as_deref(),
Some(disk),
bucket,
&path_join_buf(&[object, &src_data_dir, &format!("part.{}", part.number)]),
0,
till_offset,
erasure.shard_size(),
HashAlgorithm::HighwayHash256S,
checksum_algo.clone(),
false,
use_mmap_read,
)
.await
};
let writer = match writer_result {
Ok(writer) => writer,
Err(err) => {
writer_failure_count += 1;
all_writer_failures_retryable &= is_retryable_heal_writer_error(&err);
if first_writer_failure.is_none() {
first_writer_failure =
Some((part.number, index, heal_writer_error_summary(&err)));
first_writer_error = Some(err);
{
Ok(Some(mut reader)) => {
if let Some(proof) = &proof {
let verifier =
ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)?;
reader.set_integrity(verifier)?;
}
readers.push(Some(reader));
}
Ok(None) => {
readers.push(None);
continue;
}
Err(_e) => {
readers.push(None);
continue;
}
writers.push(None);
continue;
}
};
let mut writer = writer;
if let Some(proof) = &proof {
writer.set_integrity(
ShardVerifier::new(std::sync::Arc::clone(proof), index, 0, None)
.map_err(DiskError::from)?,
);
}
writers.push(Some(writer));
} else {
writers.push(None);
}
}
// Heal each part. erasure.Heal() will write the healed
// part to .rustfs/tmp/uuid/ which needs to be renamed
// later to the final location.
if writer_failure_count > 0
&& writers.iter().all(Option::is_none)
&& let Some(first_failure) = first_writer_failure.as_ref()
{
warn_heal_writer_failures(
bucket,
object,
version_id,
writer_failure_count,
"all_targets_unavailable",
first_failure,
);
writer_failure_warned = true;
}
let heal_part = erasure.heal(&mut writers, readers, part.size, &prefer);
let heal_part_result = if read_repair_uses_shared_lock {
match tokio::time::timeout(READ_REPAIR_DATA_PHASE_TIMEOUT, heal_part).await {
Ok(result) => result,
Err(_) => Err(DiskError::Timeout),
prefer[index] = disk.host_name().is_empty();
} else {
readers.push(None);
// errors.push(Some(DiskError::DiskNotFound));
}
}
} else {
heal_part.await
};
if let Err(e) = heal_part_result {
// Don't leak the partially-written healed shards in
// .rustfs/tmp when heal fails midway (backlog#799 B20).
let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await;
return Err(e);
}
// close_bitrot_writers(&mut writers).await?;
if !is_inline_buffer && let Some(proof) = &proof {
let before = out_dated_disks.iter().filter(|disk| disk.is_some()).count();
let proof_result = async {
let heal_part = erasure.heal(&mut writers, readers, part.size, &prefer);
let heal_part_result = if read_repair_uses_shared_lock {
match tokio::time::timeout(READ_REPAIR_DATA_PHASE_TIMEOUT, heal_part).await {
Ok(result) => result,
Err(_) => Err(DiskError::Timeout),
}
} else {
heal_part.await
};
heal_part_result?;
// close_bitrot_writers(&mut writers).await?;
if !is_inline_buffer && let Some(proof) = &proof {
let before = out_dated_disks.iter().filter(|disk| disk.is_some()).count();
PreparedIntegrity::copy_from(proof)
.await?
.write(
@@ -1746,89 +1768,71 @@ impl SetDisks {
RUSTFS_META_TMP_BUCKET,
&format!("{tmp_id}/{dst_data_dir}"),
)
.await
}
.await;
if let Err(error) = proof_result {
let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await;
return Err(error.into());
}
disks_to_heal_count -= before - out_dated_disks.iter().filter(|disk| disk.is_some()).count();
}
for (index, disk_op) in out_dated_disks.iter_mut().enumerate() {
if disk_op.is_none() {
continue;
.await?;
disks_to_heal_count -=
before - out_dated_disks.iter().filter(|disk| disk.is_some()).count();
}
if writers[index].is_none() {
*disk_op = None;
disks_to_heal_count -= 1;
continue;
}
parts_metadata[index].data_dir = Some(dst_data_dir);
parts_metadata[index].add_object_part(
part.number,
part.etag.clone(),
part.size,
part.mod_time,
part.actual_size,
part.index.clone(),
part.checksums.clone(),
);
if let Some(updated) = parts_metadata[index]
.parts
.iter_mut()
.find(|updated| updated.number == part.number)
{
updated.integrity.clone_from(&part.integrity);
}
if is_inline_buffer {
if let Some(writer) = writers[index].take() {
// if let Some(w) = writer.as_any().downcast_ref::<BitrotFileWriter>() {
// parts_metadata[index].data = Some(w.inline_data().to_vec());
// }
parts_metadata[index].data =
Some(writer.into_inline_data().map(Bytes::from).unwrap_or_default());
for (index, disk_op) in out_dated_disks.iter_mut().enumerate() {
if disk_op.is_none() {
continue;
}
parts_metadata[index].set_inline_data();
} else {
parts_metadata[index].data = None;
}
}
if disks_to_heal_count == 0 {
if !writer_failure_warned && let Some(first_failure) = first_writer_failure.as_ref() {
warn_heal_writer_failures(
bucket,
object,
version_id,
writer_failure_count,
"all_targets_unavailable",
first_failure,
if writers[index].is_none() {
*disk_op = None;
disks_to_heal_count -= 1;
continue;
}
parts_metadata[index].data_dir = Some(dst_data_dir);
parts_metadata[index].add_object_part(
part.number,
part.etag.clone(),
part.size,
part.mod_time,
part.actual_size,
part.index.clone(),
part.checksums.clone(),
);
if let Some(updated) = parts_metadata[index]
.parts
.iter_mut()
.find(|updated| updated.number == part.number)
{
updated.integrity.clone_from(&part.integrity);
}
if is_inline_buffer {
if let Some(writer) = writers[index].take() {
// if let Some(w) = writer.as_any().downcast_ref::<BitrotFileWriter>() {
// parts_metadata[index].data = Some(w.inline_data().to_vec());
// }
parts_metadata[index].data =
Some(writer.into_inline_data().map(Bytes::from).unwrap_or_default());
}
parts_metadata[index].set_inline_data();
} else {
parts_metadata[index].data = None;
}
}
// Clean up healed shards written to .rustfs/tmp before bailing (B20).
let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await;
let error = if all_writer_failures_retryable {
first_writer_error.take().unwrap_or(DiskError::FaultyDisk)
} else {
DiskError::other(format!("all drives had write errors, unable to heal {bucket}/{object}"))
};
return Ok((result, Some(error)));
}
}
if !writer_failure_warned && let Some(first_failure) = first_writer_failure.as_ref() {
warn_heal_writer_failures(
bucket,
object,
version_id,
writer_failure_count,
"partial_targets_unavailable",
first_failure,
);
if disks_to_heal_count == 0 {
writer_failures.warn(bucket, object, version_id, "all_targets_unavailable");
return Ok(Some(writer_failures.into_error()));
}
}
writer_failures.warn(bucket, object, version_id, "partial_targets_unavailable");
Ok(None)
}
.await;
match stage_result {
Ok(None) => {}
failure => {
// Release readers and writers before removing all parts
// staged under this heal's exclusively owned prefix.
let _ = self.delete_all(RUSTFS_META_TMP_BUCKET, &tmp_id).await;
return failure.map(|error| (result, error));
}
}
}
let _read_repair_write_lock_guard = if let Some(expected_fingerprint) = read_repair_commit_fingerprint {
@@ -3531,8 +3535,8 @@ impl SetDisks {
#[cfg(test)]
mod heal_result_report_tests {
use super::{
AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ReadRepairCommitFingerprint,
ReadRepairPauseScope, SetDisks, heal_writer_error_summary, is_retryable_heal_writer_error,
AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, HealWriterFailures,
ReadRepairCommitFingerprint, ReadRepairPauseScope, SetDisks, heal_writer_error_summary, is_retryable_heal_writer_error,
};
use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope};
use crate::disk::endpoint::Endpoint;
@@ -3786,10 +3790,17 @@ mod heal_result_report_tests {
#[tokio::test(flavor = "current_thread")]
#[serial_test::serial]
async fn heal_writer_failures_emit_one_aggregate_warning_per_object() {
for (case, failed_target_count, expected_result, expect_error) in [
("partial", 1usize, "partial_targets_unavailable", false),
("all", 2usize, "all_targets_unavailable", true),
for (case, target_count, failed_target_count, injected_error) in [
("partial", 2usize, 1usize, DiskError::DiskFull),
("all-full", 2, 2, DiskError::DiskFull),
("all-timeout", 2, 2, DiskError::Timeout),
("single-faulty", 1, 1, DiskError::FaultyDisk),
] {
let expected_result = if target_count == failed_target_count {
"all_targets_unavailable"
} else {
"partial_targets_unavailable"
};
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = format!("heal-writer-{case}");
let object = "object.bin";
@@ -3816,11 +3827,13 @@ mod heal_result_report_tests {
.read_version("", &bucket, object, "", &ReadOptions::default())
.await
.expect("source metadata should be readable");
let mut target_slots = [source.erasure.distribution[0] - 1, source.erasure.distribution[1] - 1];
let mut target_slots = (0..target_count)
.map(|index| source.erasure.distribution[index] - 1)
.collect::<Vec<_>>();
target_slots.sort_unstable();
for index in [0, 1] {
remove_current_object_part(&temp_dirs[index], &bucket, object)
for dir in temp_dirs.iter().take(target_count) {
remove_current_object_part(dir, &bucket, object)
.await
.expect("target shard should be removed before heal");
}
@@ -3834,7 +3847,9 @@ mod heal_result_report_tests {
.without_time()
.finish();
let subscriber_guard = tracing::subscriber::set_default(subscriber);
let failure_scope = HealWriterFailureScope::install(&bucket, object, failed_slots, DiskError::DiskFull);
let tmp_before = non_trash_tmp_entries(&temp_dirs).await;
let injected_summary = heal_writer_error_summary(&injected_error);
let failure_scope = HealWriterFailureScope::install(&bucket, object, None, failed_slots, injected_error.clone());
let heal_outcome = set
.heal_object(
@@ -3851,12 +3866,22 @@ mod heal_result_report_tests {
drop(failure_scope);
drop(subscriber_guard);
let (_, error) = heal_outcome.expect("writer setup failures must retain a heal report and their typed error");
assert_eq!(
heal_outcome.is_err(),
expect_error,
"{case}: aggregate heal result should match writer outcomes"
error,
(target_count == failed_target_count).then_some(injected_error),
"{case}: partial repair succeeds and all-target failure retains its cause"
);
assert_eq!(
non_trash_tmp_entries(&temp_dirs).await,
tmp_before,
"{case}: heal must clean temporary shards"
);
let output = logs.contents();
assert!(
!output.contains("Erasure encode write quorum unavailable"),
"{case}: unavailable heal targets must not fabricate an all-disk encode failure: {output}"
);
assert_eq!(
output.matches("Set disk object heal writer failures").count(),
1,
@@ -3879,12 +3904,172 @@ mod heal_result_report_tests {
"{case}: warning must distinguish partial from all-target failure: {output}"
);
assert!(
output.contains("error=drive path full"),
output.contains(&format!("error={injected_summary}")),
"{case}: warning must preserve a redacted failure reason: {output}"
);
}
}
#[test]
fn heal_writer_failure_aggregation_retains_permanent_cause() {
for errors in [
[DiskError::Timeout, DiskError::DiskFull],
[DiskError::DiskFull, DiskError::Timeout],
] {
let mut failures = HealWriterFailures::new(2);
for (index, error) in errors.into_iter().enumerate() {
failures.record(1, index, error);
}
assert_eq!(failures.count, 2);
assert_eq!(failures.into_error(), DiskError::DiskFull);
}
}
#[tokio::test]
#[serial_test::serial]
async fn heal_second_part_failures_clean_staged_parts_and_retry() {
use crate::set_disk::CompletePart;
use crate::storage_api_contracts::multipart::MultipartOperations as _;
use tokio::io::AsyncReadExt as _;
let (dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = "heal-writer-second-part";
let object = "multipart.bin";
for disk in &disks {
disk.make_volume(bucket).await.expect("fixture bucket");
}
let options = ObjectOptions {
no_lock: true,
..Default::default()
};
let upload = set
.new_multipart_upload(bucket, object, &options)
.await
.expect("multipart upload");
let payloads = [vec![0x35; 5 * 1024 * 1024], vec![0x69; 1024 * 1024]];
let mut completed = Vec::new();
for (index, payload) in payloads.iter().enumerate() {
let part = set
.put_object_part(
bucket,
object,
&upload.upload_id,
index + 1,
&mut PutObjReader::from_vec(payload.clone()),
&options,
)
.await
.expect("multipart part");
completed.push(CompletePart {
part_num: part.part_num,
etag: part.etag,
..Default::default()
});
}
set.clone()
.complete_multipart_upload(bucket, object, &upload.upload_id, completed, &options)
.await
.expect("completed multipart object");
let source = disks[2]
.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("healthy source metadata");
let data_dir = source.data_dir.expect("multipart data directory").to_string();
let target_slots = [source.erasure.distribution[0] - 1, source.erasure.distribution[1] - 1];
for dir in &dirs[..2] {
tokio::fs::remove_dir_all(dir.path().join(bucket).join(object).join(&data_dir))
.await
.expect("remove target shards");
}
let tmp_before = non_trash_tmp_entries(&dirs).await;
let scope = HealWriterFailureScope::install(bucket, object, Some(2), &target_slots, DiskError::Timeout);
let heal_options = HealOpts {
no_lock: true,
scan_mode: HealScanMode::Deep,
..Default::default()
};
let (_, error) = set
.heal_object(bucket, object, "", &heal_options)
.await
.expect("failed heal report");
drop(scope);
assert_eq!(error, Some(DiskError::Timeout));
assert_eq!(
non_trash_tmp_entries(&dirs).await,
tmp_before,
"first-part staging must be cleaned after the second-part setup failure"
);
let source_path = dirs[2].path().join(bucket).join(object).join(&data_dir).join("part.2");
let source_bytes = tokio::fs::read(&source_path).await.expect("committed source shard");
let mut metadata_before = Vec::new();
for dir in &dirs {
metadata_before.push(
tokio::fs::read(dir.path().join(bucket).join(object).join("xl.meta"))
.await
.expect("committed metadata before reconstruction failure"),
);
}
let (pause_scope, arrived, release) =
ReadRepairPauseScope::install_at(bucket, object, super::ReadRepairPausePhase::PartStaging(2));
let lose_source = async {
arrived.notified().await;
assert_ne!(
non_trash_tmp_entries(&dirs).await,
tmp_before,
"the first part must already be staged before the source loss"
);
tokio::fs::remove_file(&source_path)
.await
.expect("lose one remaining source shard");
release.notify_one();
};
let (outcome, ()) = tokio::time::timeout(std::time::Duration::from_secs(10), async {
tokio::join!(set.heal_object(bucket, object, "", &heal_options), lose_source)
})
.await
.expect("heal must report source loss without waiting indefinitely");
drop(pause_scope);
assert_eq!(
outcome.expect_err("source loss after writer admission must preserve the outer failure"),
DiskError::ErasureReadQuorum
);
assert_eq!(
non_trash_tmp_entries(&dirs).await,
tmp_before,
"reconstruction failure must clean every staged part"
);
for (dir, expected) in dirs.iter().zip(metadata_before) {
assert_eq!(
tokio::fs::read(dir.path().join(bucket).join(object).join("xl.meta"))
.await
.expect("metadata after reconstruction failure"),
expected,
"a failed data stage must not publish replacement metadata"
);
}
tokio::fs::write(&source_path, source_bytes)
.await
.expect("restore the committed source shard");
let (_, error) = set
.heal_object(bucket, object, "", &heal_options)
.await
.expect("retry after target recovery");
assert!(error.is_none(), "recovered targets should heal both parts: {error:?}");
let mut reader = set
.get_object_reader(bucket, object, None, Default::default(), &options)
.await
.expect("read repaired multipart object");
let mut actual = Vec::new();
reader
.stream
.read_to_end(&mut actual)
.await
.expect("complete repaired object body");
assert_eq!(actual, payloads.concat());
}
async fn real_disk() -> (TempDir, Endpoint, DiskStore) {
let dir = tempfile::tempdir().expect("tempdir should be created");
let endpoint =
crates/ecstore/src/set_disk/ops/multipart.rs
+4
View File
@@ -3004,6 +3004,10 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
// etag
let etag = resolve_complete_etag(opts, &uploaded_parts);
// An upload can contain parts from an older writer or a peer. Until
// every part's format is attested, keep its ETag out of keyless lists.
rustfs_utils::http::metadata_compat::remove_str(&mut fi.metadata, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG);
fi.metadata.insert("etag".to_owned(), etag);
let persist_encryption_original_size = should_persist_encryption_original_size(&fi.metadata);
crates/ecstore/src/set_disk/ops/object.rs
+69
View File
@@ -3847,6 +3847,16 @@ impl SetDisks {
.map(crate::io_support::rio::compression_index_storage_bytes);
let mut etag = data.stream.try_resolve_etag().unwrap_or_default();
if data.stream.encrypted_etag {
if opts.preserve_etag.as_ref().is_none_or(|preserved| preserved == &etag) {
insert_str(&mut user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
} else {
rustfs_utils::http::metadata_compat::remove_str(
&mut user_defined,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
);
}
}
if let Some(ref tag) = opts.preserve_etag {
etag = tag.clone();
}
@@ -10610,6 +10620,65 @@ mod object_encryption_resolver_wiring_tests {
use std::sync::Mutex;
use std::sync::atomic::{AtomicUsize, Ordering};
#[tokio::test]
async fn encrypted_put_attests_only_etags_produced_by_its_encryption_reader() {
use super::hermetic_set_disks_support::hermetic_set_disks_isolated;
use crate::io_support::rio::{WriteEncryption, WritePlan};
use crate::storage_api_contracts::object::ObjectIO as _;
use tokio::io::AsyncReadExt;
let (_temp_dirs, disks, set_disks) = hermetic_set_disks_isolated(4).await;
let bucket = "encrypted-etag-proof";
for disk in &disks {
disk.make_volume(bucket).await.expect("create bucket volume");
}
let plaintext = b"Salary 2026: 85000 EUR";
let size = plaintext.len() as i64;
let mut control = HashReader::from_stream(Cursor::new(plaintext), size, size, None, None, false).expect("control reader");
control.read_to_end(&mut Vec::new()).await.expect("read control plaintext");
let plaintext_md5 = control.try_resolve_etag().expect("control MD5");
for (object, preserved) in [("new", None), ("preserved", Some(plaintext_md5.clone()))] {
let reader =
HashReader::from_stream(Cursor::new(plaintext), size, size, None, None, false).expect("plaintext reader");
let reader = WritePlan::new()
.with_encryption(WriteEncryption::singlepart([1; 32], [2; 12]))
.apply(reader, size)
.expect("encrypted reader");
let mut opts = ObjectOptions {
preserve_etag: preserved.clone(),
user_defined: HashMap::from([
("x-amz-server-side-encryption-customer-algorithm".to_owned(), "AES256".to_owned()),
(rustfs_utils::http::SSEC_ORIGINAL_SIZE_HEADER.to_owned(), size.to_string()),
]),
..Default::default()
};
// A copied producer marker cannot attest to a preserved plaintext tag.
insert_str(&mut opts.user_defined, rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
set_disks
.put_object(bucket, object, &mut PutObjReader::new(reader), &opts)
.await
.expect("store encrypted bytes");
let stored = set_disks
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("read persisted metadata");
if let Some(preserved) = preserved {
assert_eq!(stored.etag.as_deref(), Some(preserved.as_str()));
assert!(!rustfs_utils::http::etag_can_be_listed(&stored.user_defined));
} else {
assert_ne!(stored.etag.as_ref(), Some(&plaintext_md5));
assert_eq!(
rustfs_utils::http::get_consistent_str(
&stored.user_defined,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG
),
Some("v1")
);
assert!(rustfs_utils::http::etag_can_be_listed(&stored.user_defined));
}
}
}
#[derive(Clone, Default)]
struct CapturedLogs(Arc<Mutex<Vec<u8>>>);
crates/ecstore/src/store/bucket.rs
+275 -25
View File
@@ -1658,34 +1658,46 @@ impl ECStore {
#[instrument(skip(self))]
pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result<Vec<BucketInfo>> {
// TODO(backlog): support cached bucket listing via opts.cached
Ok(self.list_bucket_from_sets(opts, usize::MAX).await?.buckets)
Ok(self
.list_bucket_from_sets(opts, usize::MAX, BucketInfoQuorum::Write)
.await?
.buckets)
}
pub async fn list_bucket_for_scanner(&self, opts: &BucketOptions) -> Result<crate::cluster::rpc::ScannerBucketListing> {
self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY).await
self.list_bucket_from_sets(opts, SCANNER_BUCKET_LIST_SET_CONCURRENCY, BucketInfoQuorum::Write)
.await
}
/// Discover candidates for metadata initialization at each set's namespace
/// read quorum. Metadata loading still validates persisted configuration;
/// this listing must not be used as a complete scanner or mutation view.
pub async fn list_buckets_for_metadata_startup(&self) -> Result<Vec<BucketInfo>> {
Ok(self
.list_bucket_from_sets(
&BucketOptions {
no_metadata: true,
..Default::default()
},
SCANNER_BUCKET_LIST_SET_CONCURRENCY,
BucketInfoQuorum::Read,
)
.await?
.buckets)
}
async fn list_bucket_from_sets(
&self,
opts: &BucketOptions,
max_concurrency: usize,
quorum: BucketInfoQuorum,
) -> Result<crate::cluster::rpc::ScannerBucketListing> {
let set_count = self.pools.iter().map(|pool| pool.disk_set.len()).sum();
let concurrency = bucket_list_set_concurrency(set_count, max_concurrency);
let deleted = opts.deleted;
let cached = opts.cached;
let no_metadata = opts.no_metadata;
let mut set_listings = stream::iter(self.bucket_sets().map(move |(pool_index, set_index, set)| {
let opts = BucketOptions {
deleted,
cached,
no_metadata,
};
async move {
set.list_bucket_for_scanner(&opts)
.await
.map(|(buckets, complete)| (pool_index, set_index, buckets, complete))
}
let mut set_listings = stream::iter(self.bucket_sets().map(move |(pool_index, set_index, set)| async move {
set.list_bucket_with_quorum(quorum)
.await
.map(|(buckets, complete)| (pool_index, set_index, buckets, complete))
}))
.buffer_unordered(concurrency);
let mut topology_complete = set_count != 0;
@@ -1787,6 +1799,23 @@ impl ECStore {
} else {
None
};
// Match metadata migration: lifecycle -> metadata transaction ->
// bucket namespace. Holding the transaction while waiting for the
// namespace prevents a migration writer and force delete from
// acquiring these two locks in opposite orders.
let object_lock_metadata_transaction = if opts.force && !is_meta_bucketname(bucket) {
Some(
await_bucket_namespace_operation(
bucket_lifecycle_guard.as_ref(),
bucket,
"Object Lock metadata transaction acquisition",
crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket),
)
.await?,
)
} else {
None
};
let ns_guard = if !opts.no_lock {
let ns_lock = self.new_ns_lock(bucket, bucket).await?;
Some(
@@ -1842,6 +1871,29 @@ impl ECStore {
}
}
// Keep configuration changes fenced until data deletion finishes;
// a force delete cannot bypass any Object Lock protection.
let object_lock_metadata_guard = if bucket_exists && let Some(guard) = object_lock_metadata_transaction {
let (state, incarnation, _) =
crate::bucket::metadata_sys::get_object_lock_config_and_incarnation_from_disk_with_guard_in(
&self.ctx,
bucket,
Some(ns_guard.as_ref().unwrap_or(&guard)),
)
.await?;
match state {
crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent => Some((guard, incarnation)),
crate::bucket::metadata_sys::ObjectLockConfigState::Configured { .. } => {
return Err(StorageError::PrefixAccessDenied(bucket.to_owned(), String::new()));
}
crate::bucket::metadata_sys::ObjectLockConfigState::Fabricated => {
return Err(StorageError::FileCorrupt);
}
}
} else {
None
};
if sr_delete && !bucket_exists {
delete_opts.force_if_empty = true;
}
@@ -1855,7 +1907,12 @@ impl ECStore {
ns_guard.as_ref(),
bucket,
"bucket delete marker creation",
self.mark_bucket_deleted(bucket),
await_bucket_namespace_operation(
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
bucket,
"Object Lock configuration fence during bucket retirement",
self.mark_bucket_deleted(bucket),
),
)
.await?;
}
@@ -1864,9 +1921,13 @@ impl ECStore {
// Legacy buckets without a stamp cannot produce retirement authority.
let retirement = if bucket_exists && bucket_lifecycle_guard.is_some() && !is_meta_bucketname(bucket) {
if let Some(store) = metadata_sys::object_store_if_initialized_in(&self.ctx).await {
crate::bucket::metadata::load_bucket_incarnation(store.clone(), bucket)
.await?
.map(|incarnation| (store, incarnation))
if let Some((_, incarnation)) = object_lock_metadata_guard.as_ref() {
Some((store, *incarnation))
} else {
crate::bucket::metadata::load_bucket_incarnation(store.clone(), bucket)
.await?
.map(|incarnation| (store, incarnation))
}
} else {
None
}
@@ -1878,13 +1939,19 @@ impl ECStore {
bucket_lifecycle_guard.as_ref(),
bucket,
"physical bucket deletion",
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
self.delete_bucket_on_sets(bucket, &delete_opts)
.await
.map_err(|err| to_object_err(err, vec![bucket]))
}),
await_bucket_namespace_operation(
object_lock_metadata_guard.as_ref().map(|(guard, _)| guard),
bucket,
"Object Lock configuration fence during bucket deletion",
run_physical_bucket_deletion(ns_guard.as_ref(), bucket, async {
self.delete_bucket_on_sets(bucket, &delete_opts)
.await
.map_err(|err| to_object_err(err, vec![bucket]))
}),
),
)
.await;
drop(object_lock_metadata_guard);
if let Err(err) = delete_result
&& (!sr_delete || !is_err_strict_volume_not_found(&err))
{
@@ -4414,6 +4481,79 @@ mod tests {
assert_eq!(info_err, StorageError::ErasureReadQuorum);
}
#[tokio::test]
#[serial]
async fn metadata_startup_listing_uses_each_sets_read_quorum() {
for drives_per_pool in [vec![2], vec![3], vec![4], vec![6], vec![8], vec![4, 6]] {
let (_temp_dir, store) = setup_bucket_quorum_test_env(&drives_per_pool, None).await;
let bucket = "metadata-startup-read-quorum";
for pool in &store.pools {
pool.disk_set[0]
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("healthy set should create the startup candidate");
}
let mut offline = Vec::new();
for (pool_index, &width) in drives_per_pool.iter().enumerate() {
let set = &store.pools[pool_index].disk_set[0];
let indexes = (width.div_ceil(2)..width).collect::<Vec<_>>();
offline.push(take_set_disks_offline(&store, set, &indexes).await);
}
let listing = store
.list_buckets_for_metadata_startup()
.await
.expect("startup must discover buckets at every set's read quorum");
assert_eq!(listing.len(), 1);
assert_eq!(listing[0].name, bucket);
if drives_per_pool.iter().any(|width| width % 2 == 0) {
let error = store
.list_bucket_for_scanner(&BucketOptions::default())
.await
.expect_err("startup discovery must not weaken scanner quorum");
assert_eq!(error, StorageError::ErasureWriteQuorum);
let error = store
.list_bucket(&BucketOptions {
no_metadata: true,
..Default::default()
})
.await
.expect_err("ordinary listing must retain its existing quorum");
assert_eq!(error, StorageError::ErasureWriteQuorum);
}
// A healthy earlier pool must not hide read-quorum loss in a later pool.
let last_pool = store.pools.len() - 1;
let set = &store.pools[last_pool].disk_set[0];
let extra = take_set_disks_offline(&store, set, &[0]).await;
let error = store
.list_buckets_for_metadata_startup()
.await
.expect_err("every set must retain read quorum for startup discovery");
assert_eq!(error, StorageError::ErasureReadQuorum);
restore_set_disks(&store, set, extra).await;
for (pool_index, disks) in offline.into_iter().enumerate() {
restore_set_disks(&store, &store.pools[pool_index].disk_set[0], disks).await;
}
}
}
#[tokio::test]
#[serial]
async fn metadata_startup_listing_does_not_accept_single_disk_bucket_remnants() {
let (temp_dir, store) = setup_bucket_quorum_test_env(&[4], None).await;
let bucket = "incomplete-startup-bucket";
tokio::fs::create_dir(temp_dir.path().join("pool0-disk0").join(bucket))
.await
.expect("create one uncommitted directory remnant");
let set = &store.pools[0].disk_set[0];
let offline = take_set_disks_offline(&store, set, &[2, 3]).await;
let listing = store
.list_buckets_for_metadata_startup()
.await
.expect("two responsive disks satisfy discovery quorum");
assert!(listing.iter().all(|info| info.name != bucket));
restore_set_disks(&store, set, offline).await;
}
#[tokio::test]
#[serial]
async fn scanner_bucket_listing_unions_every_erasure_set() {
@@ -4628,6 +4768,116 @@ mod tests {
assert!(!any_disk_path_exists(&disk_paths, bucket_deleted_marker_volume(&bucket)).await);
}
#[tokio::test]
#[serial]
async fn force_bucket_delete_preserves_object_lock_versions() {
let (_temp_dir, store) = setup_bucket_quorum_test_env(&[4], Some(2)).await;
metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = format!("force-delete-worm-{}", Uuid::new_v4().simple());
store
.make_bucket(
&bucket,
&MakeBucketOptions {
lock_enabled: true,
versioning_enabled: true,
..Default::default()
},
)
.await
.expect("create Object Lock bucket");
let until = (OffsetDateTime::now_utc() + time::Duration::days(365))
.format(&time::format_description::well_known::Rfc3339)
.expect("format future retention date");
let written = store
.put_object(
&bucket,
"retained",
&mut PutObjReader::from_vec(b"retained bytes".to_vec()),
&ObjectOptions {
versioned: true,
user_defined: HashMap::from([
("x-amz-object-lock-mode".to_owned(), "COMPLIANCE".to_owned()),
("x-amz-object-lock-retain-until-date".to_owned(), until),
]),
..Default::default()
},
)
.await
.expect("write retained object");
for operation in [SRBucketDeleteOp::NoOp, SRBucketDeleteOp::MarkDelete, SRBucketDeleteOp::Purge] {
let err = store
.delete_bucket(
&bucket,
&DeleteBucketOptions {
force: true,
srdelete_op: operation,
..Default::default()
},
)
.await
.expect_err("force deletion must not bypass Object Lock");
assert!(matches!(err, StorageError::PrefixAccessDenied(_, _)), "unexpected denial: {err}");
let info = store
.get_object_info(
&bucket,
"retained",
&ObjectOptions {
version_id: written.version_id.map(|version| version.to_string()),
..Default::default()
},
)
.await
.expect("retained version must survive the rejected bucket deletion");
assert_eq!(info.etag, written.etag);
assert_eq!(info.size, written.size);
assert_eq!(info.version_id, written.version_id);
}
metadata_sys::inject_object_lock_disk_read_error_in(&store.ctx, &bucket)
.await
.expect("inject authoritative metadata read failure");
let error = store
.delete_bucket(
&bucket,
&DeleteBucketOptions {
force: true,
..Default::default()
},
)
.await
.expect_err("an unreadable Object Lock configuration must fail closed");
assert!(error.to_string().contains("injected Object Lock metadata disk read failure"));
let info = store
.get_object_info(&bucket, "retained", &ObjectOptions::default())
.await
.expect("metadata read failure must leave the retained object intact");
assert_eq!(info.version_id, written.version_id);
let unlocked = format!("force-delete-unlocked-{}", Uuid::new_v4().simple());
store
.make_bucket(&unlocked, &MakeBucketOptions::default())
.await
.expect("create unlocked control bucket");
store
.put_object(
&unlocked,
"ordinary",
&mut PutObjReader::from_vec(b"ordinary bytes".to_vec()),
&ObjectOptions::default(),
)
.await
.expect("write ordinary control object");
store
.delete_bucket(
&unlocked,
&DeleteBucketOptions {
force: true,
..Default::default()
},
)
.await
.expect("force deletion must still work for an unlocked bucket");
}
#[tokio::test]
#[serial]
async fn bucket_delete_mark_delete_rejects_hidden_object_paths_without_force() {
crates/ecstore/src/store/init.rs
+30 -3
View File
@@ -108,6 +108,10 @@ fn should_retry_format_load(err: &Error) -> bool {
)
}
fn format_load_retry_exhausted(err: &Error, retries: u32) -> bool {
retries >= 10 && !matches!(err, Error::ErasureReadQuorum | Error::FirstDiskWait | Error::NotFirstDisk)
}
fn should_auto_start_rebalance_after_init(decommission_running: bool, rebalance_resume_required: bool) -> bool {
rebalance_resume_required && !decommission_running
}
@@ -484,7 +488,7 @@ impl ECStore {
check_disk_fatal_errs(&errs)?;
let loaded_format = {
let mut times = 0;
let mut times = 0_u32;
let mut interval = 1;
loop {
match init_format::connect_load_init_formats_with_instance_ctx(
@@ -500,13 +504,13 @@ impl ECStore {
Ok(fm) => break Ok(fm),
Err(e) if !should_retry_format_load(&e) => break Err(e),
// Wrap the final error if we are giving up
Err(e) if times >= 10 => {
Err(e) if format_load_retry_exhausted(&e, times) => {
break Err(Error::other(format!("store init failed to load formats after {times} retries: {e}")));
}
// Retrying so just drop the error
Err(_) => {}
}
times += 1;
times = times.saturating_add(1);
if interval < 16 {
interval *= 2;
}
@@ -519,6 +523,12 @@ impl ECStore {
"Retrying storage format load"
);
select! {
_ = ctx.cancelled() => {
return Err(Error::other(std::io::Error::new(
std::io::ErrorKind::Interrupted,
"storage format initialization cancelled",
)));
}
_ = tokio::signal::ctrl_c() => {
info!(
event = EVENT_STORE_FORMAT_RETRY,
@@ -2201,6 +2211,23 @@ mod tests {
assert_eq!(attempts.load(Ordering::SeqCst), 2);
}
#[test]
fn format_peer_wait_does_not_exhaust_the_retry_budget() {
for retries in [0, 9, 10, u32::MAX] {
for error in [
StorageError::ErasureReadQuorum,
StorageError::FirstDiskWait,
StorageError::NotFirstDisk,
] {
assert!(
!super::format_load_retry_exhausted(&error, retries),
"peer availability must remain retryable"
);
}
assert_eq!(super::format_load_retry_exhausted(&StorageError::Timeout, retries), retries >= 10);
}
}
#[test]
fn test_should_retry_format_load_rejects_permanent_corruption() {
assert!(!should_retry_format_load(&StorageError::CorruptedFormat));
crates/ecstore/src/store/init_format.rs
+233
View File
@@ -134,6 +134,18 @@ pub(crate) async fn connect_load_init_formats_with_instance_ctx(
{
return Err(Error::CorruptedFormat);
}
if matches!(format_quorum, Some(Err(Error::ErasureReadQuorum)))
&& let Some((format, members)) = select_existing_format_read_quorum(&formats, &errs, set_drive_count)
{
retain_format_quorum_members(instance_ctx, disks, &format, &members, set_drive_count).await?;
// Loading an existing format grants no authority to initialize pool
// metadata or migrate formats. The persisted storage classes and each
// object's EC metadata still determine service and operation quorums.
return Ok(LoadedFormat {
format,
pool_meta_bootstrap_authority: PoolMetaBootstrapAuthority::None,
});
}
let resumable_partial_migration = formats_present > 0
&& errs.iter().any(|error| matches!(error, Some(DiskError::UnformattedDisk)))
&& format_quorum.as_ref().is_some_and(Result::is_err)
@@ -578,6 +590,43 @@ pub fn get_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: u
select_format_erasure_in_quorum(formats, slot_offset).map(|(format, _)| format)
}
/// Load a consistent existing layout with enough members per set to read the
/// max-parity system metadata containing the persisted storage classes. This
/// is not object-read admission: STANDARD, RRS and historical objects can have
/// different EC layouts, which are validated after this bootstrap read.
/// Format migration and healing still use strict-majority election.
fn select_existing_format_read_quorum(
formats: &[Option<FormatV3>],
errors: &[Option<DiskError>],
set_drive_count: usize,
) -> Option<(FormatV3, Vec<bool>)> {
if set_drive_count == 0 || formats.is_empty() || formats.len() != errors.len() {
return None;
}
let metadata_parity = set_drive_count / 2;
let metadata_read_quorum = set_drive_count - metadata_parity;
let reference = formats.iter().flatten().next()?;
if !formats_match_reference_slots(formats, reference, 0)
|| check_format_erasure_value_for_topology(reference, formats.len(), set_drive_count).is_err()
|| formats.iter().zip(errors).any(|(format, error)| match (format, error) {
(Some(_), None) => false,
(None, Some(DiskError::DiskNotFound)) => false,
(None, Some(error)) => !is_network_like_disk_error(error),
_ => true,
})
|| formats
.chunks(set_drive_count)
.any(|set| set.iter().flatten().count() < metadata_read_quorum)
{
return None;
}
let mut format = reference.clone();
format.erasure.this = Uuid::nil();
format.disk_info = None;
Some((format, formats.iter().map(Option::is_some).collect()))
}
pub(crate) fn select_format_erasure_in_quorum(formats: &[Option<FormatV3>], slot_offset: usize) -> Result<(FormatV3, Vec<bool>)> {
let mut candidates = HashMap::new();
let formats_present = formats.iter().flatten().count();
@@ -1120,6 +1169,190 @@ mod tests {
assert!(!formats_match_reference_slots(&formats, &reference, 0));
}
fn half_pool_formats() -> (Vec<Option<FormatV3>>, Vec<Option<DiskError>>) {
let reference = FormatV3::new(6, 4);
let formats = (0..24)
.map(|slot| {
(slot % 4 < 2).then(|| {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
format
})
})
.collect::<Vec<_>>();
let errors = formats
.iter()
.map(|format| format.is_none().then_some(DiskError::DiskNotFound))
.collect();
(formats, errors)
}
#[test]
fn existing_format_read_quorum_follows_each_sets_metadata_layout() {
// These are system-metadata EC boundaries, independent of user-object
// storage classes and of how many drives are hosted by each node.
for (set_width, read_quorum) in [(1, 1), (2, 1), (3, 2), (4, 2), (6, 3), (8, 4), (12, 6), (16, 8)] {
for set_count in [1, 2, 5] {
let reference = FormatV3::new(set_count, set_width);
let mut formats = (0..set_count * set_width)
.map(|slot| {
let set = slot / set_width;
let drive = slot % set_width;
// Rotate surviving slots between sets so admission
// cannot depend on the first drive or node being up.
((drive + set) % set_width < read_quorum).then(|| {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[set][drive];
format
})
})
.collect::<Vec<_>>();
let mut errors = formats
.iter()
.map(|format| format.is_none().then_some(DiskError::DiskNotFound))
.collect::<Vec<_>>();
let (loaded, members) = select_existing_format_read_quorum(&formats, &errors, set_width)
.unwrap_or_else(|| panic!("metadata read quorum must load {set_count} sets of {set_width} drives"));
assert_eq!(loaded.shared_identity(), reference.shared_identity());
assert!(loaded.erasure.this.is_nil());
assert_eq!(members.iter().filter(|member| **member).count(), set_count * read_quorum);
// More members in other sets cannot compensate for one set
// falling below its own metadata reconstruction boundary.
for slot in 0..(set_count - 1) * set_width {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / set_width][slot % set_width];
formats[slot] = Some(format);
errors[slot] = None;
}
let lost_slot = formats
.iter()
.rposition(Option::is_some)
.expect("at least one surviving drive");
formats[lost_slot] = None;
errors[lost_slot] = Some(DiskError::DiskNotFound);
assert!(
select_existing_format_read_quorum(&formats, &errors, set_width).is_none(),
"one set below read quorum must block {set_count} sets of {set_width} drives"
);
}
}
}
#[test]
fn existing_format_read_quorum_preserves_election_and_set_boundaries() {
let (mut formats, mut errors) = half_pool_formats();
assert!(select_existing_format_read_quorum(&[], &[], 0).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors[..23], 4).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 0).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 6).is_none());
assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_some());
assert!(matches!(get_format_erasure_in_quorum(&formats, 0), Err(Error::ErasureReadQuorum)));
let mut extra = formats[0].as_ref().expect("seed format").clone();
extra.erasure.this = extra.erasure.sets[0][2];
formats[2] = Some(extra);
errors[2] = None;
assert!(get_format_erasure_in_quorum(&formats, 0).is_ok(), "13 votes must establish a majority");
formats[5] = None;
errors[5] = Some(DiskError::DiskNotFound);
assert!(
select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
"12 votes cannot compensate for a set with only one readable drive"
);
formats[2] = None;
errors[2] = Some(DiskError::DiskNotFound);
assert!(select_existing_format_read_quorum(&formats, &errors, 4).is_none(), "11 votes must wait");
}
#[test]
fn existing_format_read_quorum_rejects_conflicts_invalid_slots_and_blank_disks() {
for failure in [
"conflict",
"wrong_slot",
"nil",
"duplicate",
"blank",
"corrupt",
"access_denied",
"tie",
] {
let (mut formats, mut errors) = half_pool_formats();
match failure {
"conflict" => formats[1].as_mut().expect("second format").id = Uuid::new_v4(),
"wrong_slot" => {
let first_id = formats[0].as_ref().expect("first format").erasure.this;
formats[1].as_mut().expect("second format").erasure.this = first_id;
}
"nil" => {
for format in formats.iter_mut().flatten() {
format.id = Uuid::nil();
}
}
"duplicate" => {
for format in formats.iter_mut().flatten() {
format.erasure.sets[5][3] = format.erasure.sets[0][0];
}
}
"blank" => errors[2] = Some(DiskError::UnformattedDisk),
"corrupt" => errors[2] = Some(DiskError::CorruptedFormat),
"access_denied" => errors[2] = Some(DiskError::FileAccessDenied),
"tie" => {
let other = FormatV3::new(6, 4);
for slot in 0..24 {
if formats[slot].is_none() {
let mut format = other.clone();
format.erasure.this = other.erasure.sets[slot / 4][slot % 4];
formats[slot] = Some(format);
errors[slot] = None;
}
}
}
_ => unreachable!(),
}
assert!(
select_existing_format_read_quorum(&formats, &errors, 4).is_none(),
"{failure} must not authorize degraded startup"
);
}
}
#[tokio::test]
async fn existing_format_load_accepts_each_half_of_four_by_six_without_format_writes() {
let (_root, all_disks) = local_disks_with_set_width(24, 4).await;
let reference = FormatV3::new(6, 4);
for (slot, disk) in all_disks.iter().enumerate() {
let mut format = reference.clone();
format.erasure.this = reference.erasure.sets[slot / 4][slot % 4];
save_format_file(disk, &Some(format)).await.expect("persist existing format");
}
let before = format_bytes(&all_disks).await;
for first in 0..4 {
for second in first + 1..4 {
let mut disks = all_disks.clone();
for (slot, disk) in disks.iter_mut().enumerate() {
if slot % 4 != first && slot % 4 != second {
*disk = None;
}
}
let loaded = connect_load_init_formats_with_instance_ctx(
&Arc::new(InstanceContext::new()),
first == 0,
&mut disks,
6,
4,
None,
)
.await
.expect("two nodes must load the existing format");
assert_eq!(loaded.format.shared_identity(), reference.shared_identity());
assert!(matches!(loaded.pool_meta_bootstrap_authority, PoolMetaBootstrapAuthority::None));
assert_eq!(disks.iter().flatten().count(), 12);
assert_eq!(format_bytes(&all_disks).await, before, "read startup must never rewrite formats");
}
}
}
#[tokio::test]
async fn existing_format_load_succeeds_with_a_strict_majority() {
let (_temp_dir, mut disks) = two_local_disks_with_missing_third().await;
crates/ecstore/src/store/list_objects.rs
+116 -46
View File
@@ -448,8 +448,8 @@ const MAX_LIST_OBJECTS_METADATA_FAST_STALENESS_MS: u64 = 60_000;
const LIST_OBJECTS_INDEX_PROVIDER_WALKER_KEY_ONLY: &str = "walker_key_only";
const LIST_OBJECTS_INDEX_PROVIDER_PERSISTENT_KEY_ONLY: &str = "persistent_key_only";
const LIST_OBJECTS_INDEX_PROVIDER_PERSISTENT_KEY_ONLY_DEFAULT_GENERATION: &str = "persistent-key-only";
const PERSISTENT_KEY_ONLY_INDEX_FORMAT_VERSION: u8 = 2;
const PERSISTENT_KEY_ONLY_INDEX_HEADER: &str = "# rustfs-listobjects-key-only-v2";
const PERSISTENT_KEY_ONLY_INDEX_FORMAT_VERSION: u8 = 3;
const PERSISTENT_KEY_ONLY_INDEX_HEADER: &str = "# rustfs-listobjects-key-only-v3";
const PERSISTENT_KEY_ONLY_INDEX_BUCKET_HEADER: &str = "# bucket=";
const PERSISTENT_KEY_ONLY_INDEX_GENERATION_HEADER: &str = "# generation=";
const PERSISTENT_KEY_ONLY_INDEX_CHECKPOINT_HEADER: &str = "# checkpoint_high_water_mark=";
@@ -656,7 +656,10 @@ impl PersistentListMetadataObject {
name: object.name.clone(),
size: object.size,
mod_time: object.mod_time,
etag: object.etag.clone(),
etag: object
.etag
.clone()
.filter(|_| rustfs_utils::http::etag_can_be_listed(&object.user_defined)),
storage_class: object.storage_class.clone(),
}
}
@@ -2361,23 +2364,43 @@ fn list_objects_paginate(
(objects, prefixes, is_truncated, next_marker, next_version_idmarker)
}
fn list_objects_paginate_versions<'a>(
mut get_objects: Vec<ObjectInfo>,
delimiter: &Option<String>,
struct VersionedListPagination<'a, I> {
delimiter: &'a Option<String>,
max_keys: i32,
disk_has_more: bool,
cache_id: Option<&str>,
marker: Option<&str>,
(last_scanned_key, raw_keys): (Option<&str>, impl Iterator<Item = &'a str>),
) -> (Vec<ObjectInfo>, Vec<String>, bool, Option<String>, Option<String>) {
filter_versions_common_prefixes_after_marker(&mut get_objects, delimiter.as_deref(), marker);
let prefix_at_page_boundary =
versioned_prefix_page_boundary_key(&get_objects, raw_keys, delimiter.as_deref(), max_keys, disk_has_more);
cache_id: Option<&'a str>,
marker: Option<&'a str>,
last_scanned_key: Option<&'a str>,
raw_keys: I,
}
let (objects, prefixes, is_truncated, mut next_marker, mut next_version_idmarker) =
list_objects_paginate(get_objects, delimiter, max_keys, disk_has_more, cache_id, true, last_scanned_key);
fn list_objects_paginate_versions<'a, I>(
mut get_objects: Vec<ObjectInfo>,
pagination: VersionedListPagination<'a, I>,
) -> (Vec<ObjectInfo>, Vec<String>, bool, Option<String>, Option<String>)
where
I: Iterator<Item = &'a str>,
{
filter_versions_common_prefixes_after_marker(&mut get_objects, pagination.delimiter.as_deref(), pagination.marker);
let prefix_at_page_boundary = versioned_prefix_page_boundary_key(
&get_objects,
pagination.raw_keys,
pagination.delimiter.as_deref(),
pagination.max_keys,
pagination.disk_has_more,
);
let (objects, prefixes, is_truncated, mut next_marker, mut next_version_idmarker) = list_objects_paginate(
get_objects,
pagination.delimiter,
pagination.max_keys,
pagination.disk_has_more,
pagination.cache_id,
true,
pagination.last_scanned_key,
);
if is_truncated && let Some(raw_key) = prefix_at_page_boundary {
next_marker = Some(append_list_cache_id_to_marker(raw_key, cache_id));
next_marker = Some(append_list_cache_id_to_marker(raw_key, pagination.cache_id));
next_version_idmarker = None;
}
@@ -4325,12 +4348,15 @@ impl ECStore {
let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate_versions(
get_objects,
&delimiter,
max_keys,
disk_has_more,
next_cache_id.as_deref(),
opts.marker.as_deref(),
(last_scanned_key.as_deref(), entries.entries().iter().map(|entry| entry.name.as_str())),
VersionedListPagination {
delimiter: &delimiter,
max_keys,
disk_has_more,
cache_id: next_cache_id.as_deref(),
marker: opts.marker.as_deref(),
last_scanned_key: last_scanned_key.as_deref(),
raw_keys: entries.entries().iter().map(|entry| entry.name.as_str()),
},
);
Ok(ListObjectVersionsInfo {
@@ -5752,12 +5778,15 @@ impl Sets {
let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate_versions(
get_objects,
&delimiter,
max_keys,
disk_has_more,
next_cache_id.as_deref(),
opts.marker.as_deref(),
(last_scanned_key.as_deref(), entries.entries().iter().map(|entry| entry.name.as_str())),
VersionedListPagination {
delimiter: &delimiter,
max_keys,
disk_has_more,
cache_id: next_cache_id.as_deref(),
marker: opts.marker.as_deref(),
last_scanned_key: last_scanned_key.as_deref(),
raw_keys: entries.entries().iter().map(|entry| entry.name.as_str()),
},
);
Ok(ListObjectVersionsInfo {
@@ -6565,12 +6594,15 @@ impl SetDisks {
let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) = list_objects_paginate_versions(
get_objects,
&delimiter,
max_keys,
disk_has_more,
next_cache_id.as_deref(),
opts.marker.as_deref(),
(last_scanned_key.as_deref(), entries.entries().iter().map(|entry| entry.name.as_str())),
VersionedListPagination {
delimiter: &delimiter,
max_keys,
disk_has_more,
cache_id: next_cache_id.as_deref(),
marker: opts.marker.as_deref(),
last_scanned_key: last_scanned_key.as_deref(),
raw_keys: entries.entries().iter().map(|entry| entry.name.as_str()),
},
);
Ok(ListObjectVersionsInfo {
@@ -8844,12 +8876,15 @@ mod test {
let get_objects = fold_delimiter_page(&window, "", "-");
let (objects, prefixes, is_truncated, next_marker, _v) = super::list_objects_paginate_versions(
get_objects,
&delimiter,
max_keys,
disk_has_more,
None,
marker.as_deref(),
(last_scanned.as_deref(), window.iter().map(String::as_str)),
super::VersionedListPagination {
delimiter: &delimiter,
max_keys,
disk_has_more,
cache_id: None,
marker: marker.as_deref(),
last_scanned_key: last_scanned.as_deref(),
raw_keys: window.iter().map(String::as_str),
},
);
assert!(
@@ -8922,12 +8957,15 @@ mod test {
let get_objects = fold_delimiter_page(&window, "", "-");
let (objects, prefixes, is_truncated, next_marker, _) = super::list_objects_paginate_versions(
get_objects,
&delimiter,
max_keys,
disk_has_more,
None,
marker.as_deref(),
(last_scanned.as_deref(), window.iter().map(String::as_str)),
super::VersionedListPagination {
delimiter: &delimiter,
max_keys,
disk_has_more,
cache_id: None,
marker: marker.as_deref(),
last_scanned_key: last_scanned.as_deref(),
raw_keys: window.iter().map(String::as_str),
},
);
assert_eq!(objects.len() + prefixes.len(), 1, "a page must respect max-keys");
@@ -9170,6 +9208,38 @@ mod test {
assert_eq!(parsed, object);
}
#[test]
fn persistent_listing_snapshots_cannot_restore_legacy_encrypted_fingerprints() {
let mut object = ObjectInfo {
name: "encrypted".to_owned(),
etag: Some("fbfcbcc6d2f035411e6268c8b119593e".to_owned()),
user_defined: Arc::new(HashMap::from([(
"x-amz-server-side-encryption-customer-algorithm".to_owned(),
"AES256".to_owned(),
)])),
..Default::default()
};
let snapshot = PersistentListMetadataObject::from_object_info(&object);
assert!(snapshot.to_object_info("bucket").etag.is_none());
let mut old_snapshot = snapshot;
old_snapshot.etag = object.etag.clone();
let old_contents = format!(
"# rustfs-listobjects-key-only-v2\n{}\n",
encode_persistent_list_metadata_object(&old_snapshot)
);
let old_index = parse_persistent_key_only_index(&old_contents);
assert_ne!(old_index.format_version, PERSISTENT_KEY_ONLY_INDEX_FORMAT_VERSION);
rustfs_utils::http::metadata_compat::insert_str(
Arc::make_mut(&mut object.user_defined),
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
"v1".to_owned(),
);
assert_eq!(PersistentListMetadataObject::from_object_info(&object).etag, object.etag);
object.user_defined = Arc::new(HashMap::new());
assert_eq!(PersistentListMetadataObject::from_object_info(&object).etag, object.etag);
}
#[test]
fn persistent_key_only_index_parses_legacy_keys_and_metadata_snapshots() {
let object = PersistentListMetadataObject {
crates/ecstore/src/store/object.rs
+83 -3
View File
@@ -5351,13 +5351,41 @@ impl ECStore {
tier_journal_api: Option<Arc<ECStore>>,
) -> (Vec<DeletedObject>, Vec<Option<Error>>, Vec<Option<DeleteAccounting>>) {
Box::pin(async move {
let count = objects.len();
let mut deleted = vec![DeletedObject::default(); count];
let mut errors: Vec<Option<Error>> = (0..count).map(|_| None).collect();
let mut accounting = vec![None; count];
let mut valid_objects = Vec::with_capacity(count);
let mut valid_indices = Vec::with_capacity(count);
// Validate the literal key before encoding it or taking locks. A
// dot-segment key must never reach filesystem path resolution.
for (index, object) in objects.into_iter().enumerate() {
match check_del_obj_args(bucket, &object.object_name) {
Ok(()) => {
valid_indices.push(index);
valid_objects.push(object);
}
Err(err) => errors[index] = Some(err),
}
}
if valid_objects.is_empty() {
return (deleted, errors, accounting);
}
let mut opts = opts;
let receipt_sink = install_tier_free_version_receipt_sink(&mut opts);
let result = self
.handle_delete_objects_with_journal_and_accounting_inner(bucket, objects, opts, tier_journal_api)
let (valid_deleted, valid_errors, valid_accounting) = self
.handle_delete_objects_with_journal_and_accounting_inner(bucket, valid_objects, opts, tier_journal_api)
.await;
enqueue_recorded_tier_free_versions(self, receipt_sink).await;
result
for (index, ((object, error), receipt)) in valid_indices
.into_iter()
.zip(valid_deleted.into_iter().zip(valid_errors).zip(valid_accounting))
{
deleted[index] = object;
errors[index] = error;
accounting[index] = receipt;
}
(deleted, errors, accounting)
})
.await
}
@@ -9094,6 +9122,58 @@ mod tests {
(dirs, store)
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn batch_delete_rejects_dot_segments_and_keeps_valid_result_positions() {
let bucket = "batch-delete-key-validation";
let (_dirs, store) = multipool_version_test_store(bucket).await;
let options = ObjectOptions {
versioned: true,
..Default::default()
};
let victim = store
.put_object(bucket, "victim/object", &mut PutObjReader::from_vec(b"victim".to_vec()), &options)
.await
.expect("write protected-prefix control object");
store
.put_object(bucket, "allowed/object", &mut PutObjReader::from_vec(b"allowed".to_vec()), &options)
.await
.expect("write valid deletion control object");
let keys = [
"allowed/../victim/object",
"allowed/object",
"a/./b",
"../other-bucket/object",
];
let objects = keys
.iter()
.map(|key| ObjectToDelete {
object_name: (*key).to_owned(),
..Default::default()
})
.collect();
let (deleted, errors) = store.handle_delete_objects(bucket, objects, options).await;
assert_eq!(deleted.len(), keys.len());
assert_eq!(errors.len(), keys.len());
for index in [0, 2, 3] {
assert!(
matches!(&errors[index], Some(StorageError::ObjectNameInvalid(_, _))),
"invalid key {}: {:?}",
keys[index],
errors[index]
);
assert!(!deleted[index].delete_marker);
}
assert!(errors[1].is_none(), "valid deletion must still succeed: {:?}", errors[1]);
assert!(deleted[1].delete_marker);
assert_eq!(deleted[1].object_name, "allowed/object");
let after = store
.get_object_info(bucket, "victim/object", &ObjectOptions::default())
.await
.expect("the resolved victim path must remain visible");
assert_eq!(after.version_id, victim.version_id);
assert_eq!(after.etag, victim.etag);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn retired_marker_converges_after_bucket_recreation_on_sixteen_disks() {
let bucket = "retired-marker-c11";
crates/heal/tests/heal_concurrent_delete_test.rs
+24 -2
View File
@@ -563,9 +563,31 @@ fn admin_root_completes_after_unversioned_delete() {
run_delete_case(DeleteCase::Unversioned);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[test]
#[serial]
async fn admin_root_rejects_recreated_bucket_before_successor_repair() {
fn admin_root_rejects_recreated_bucket_before_successor_repair() {
// Incarnation-bound repairs use a spawned owner; retain the server's stack
// budget for the whole scenario. block_on alone would poll that future on
// libtest's smaller calling-thread stack.
let stack_size = 8 * 1024 * 1024;
std::thread::Builder::new()
.name("heal-concurrent-delete".to_owned())
.stack_size(stack_size)
.spawn(move || {
let runtime = tokio::runtime::Builder::new_multi_thread()
.worker_threads(4)
.thread_stack_size(stack_size)
.enable_all()
.build()
.expect("concurrent delete runtime");
runtime.block_on(recreated_bucket_repair());
})
.expect("concurrent delete scenario thread starts")
.join()
.expect("concurrent delete scenario must not panic");
}
async fn recreated_bucket_repair() {
let directory = tempfile::tempdir().expect("recreated bucket fixture");
let env = TestECStoreEnv::builder().base_dir(directory.path()).build().await;
env.make_bucket(BUCKET, false).await;
crates/iam/Cargo.toml
+2
View File
@@ -30,6 +30,7 @@ workspace = true
[features]
default = []
test-util = []
hotpath = [
"hotpath/hotpath",
"hotpath/tokio",
@@ -124,6 +125,7 @@ rustls-pki-types.workspace = true
serial_test = { workspace = true }
temp-env = { workspace = true, features = ["async_closure"] }
tempfile = { workspace = true }
tracing-subscriber = { workspace = true, features = ["fmt"] }
[lib]
doctest = false
crates/iam/src/federation/mod.rs
+7 -8
View File
@@ -16,21 +16,20 @@ mod binding;
mod error;
mod mapper;
mod model;
pub mod oidc;
mod provider;
mod registry;
mod transaction;
pub use binding::FederatedSessionBinding;
pub use error::{FederatedSessionBindingError, FederationError, Result};
pub(crate) use mapper::FederatedAuthorizationRule;
#[cfg(test)]
pub(crate) use mapper::FederatedAuthorizationRuleRef;
pub use mapper::{CoreFederatedAuthorizationMapper, FederatedAuthorizationRules};
pub(crate) use mapper::{FederatedAuthorizationRule, FederatedAuthorizationRuleRef};
pub(crate) use model::OpaqueLogoutContinuation;
pub use model::{
FederatedAuthorization, FederatedClaims, FederatedCodeExchange, FederatedLoginSession, FederatedProviderRef,
FederatedSession, FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, VerifiedFederatedIdentity,
FederatedAuthorization, FederatedClaims, FederatedLoginSession, FederatedProviderRef, FederatedSession,
FederatedSessionTransaction, OIDC_VIRTUAL_PARENT_CLAIM, VerifiedFederatedIdentity,
};
pub use provider::{FederatedIdentityProvider, FederatedProviderView, FederatedRedirectPolicy};
pub(crate) use provider::{StandardOidcAuthentication, VerifiedFederatedCodeExchange};
pub use registry::FederatedIdentityRegistry;
pub(crate) use provider::{FederatedProviderQuery, StandardOidcAuthentication, VerifiedFederatedCodeExchange};
pub use provider::{FederatedProviderView, FederatedRedirectPolicy};
pub use transaction::FederatedIdentityService;
crates/iam/src/federation/model.rs
+2 -50
View File
@@ -164,40 +164,6 @@ impl FederatedAuthorization {
}
}
pub struct FederatedCodeExchange {
pub authorization: FederatedAuthorization,
pub redirect_after: Option<String>,
pub id_token: String,
}
impl FederatedCodeExchange {
pub(crate) fn into_parts(self) -> (FederatedAuthorization, Option<String>, OpaqueLogoutContinuation) {
let Self {
authorization,
redirect_after,
id_token,
} = self;
let continuation = OpaqueLogoutContinuation {
provider: FederatedProviderRef::new(authorization.provider_id.clone()),
id_token,
};
(authorization, redirect_after, continuation)
}
}
impl fmt::Debug for FederatedCodeExchange {
fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result {
formatter
.debug_struct("FederatedCodeExchange")
.field("provider_id", &self.authorization.provider_id)
.field("policy_count", &self.authorization.policies.len())
.field("group_count", &self.authorization.groups.len())
.field("redirect_after_present", &self.redirect_after.is_some())
.field("id_token_present", &!self.id_token.is_empty())
.finish()
}
}
/// Carries the provider-bound ID token between code exchange and logout token creation.
pub(crate) struct OpaqueLogoutContinuation {
provider: FederatedProviderRef,
@@ -397,27 +363,13 @@ mod tests {
#[test]
fn logout_continuation_binds_provider_and_redacts_token() {
let mut authorization = authorization(Vec::new(), Vec::new());
authorization
.claims
.raw
.insert("attribute_poison".to_string(), Value::String("secret-attribute".to_string()));
let exchange = FederatedCodeExchange {
authorization,
redirect_after: Some("/console?poison=redirect-secret".to_string()),
id_token: "secret-id-token".to_string(),
};
let debug = format!("{exchange:?}");
let (_, _, continuation) = exchange.into_parts();
let continuation =
OpaqueLogoutContinuation::new(FederatedProviderRef::new("standard_oidc".to_string()), "secret-id-token".to_string());
let continuation_debug = format!("{continuation:?}");
let (provider, id_token) = continuation.into_parts();
assert_eq!(provider.as_str(), "standard_oidc");
assert_eq!(id_token, "secret-id-token");
assert!(!debug.contains("secret-id-token"));
assert!(!debug.contains("secret-attribute"));
assert!(!debug.contains("redirect-secret"));
assert!(debug.contains("id_token_present: true"));
assert!(!continuation_debug.contains("secret-id-token"));
assert!(continuation_debug.contains("[REDACTED]"));
}
crates/iam/src/federation/oidc/claims.rs
-68
View File
@@ -1,68 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{
federation::{FederatedClaims, FederatedProviderRef, VerifiedFederatedIdentity},
oidc::OidcClaims,
};
pub(super) fn verified_identity(provider: FederatedProviderRef, claims: OidcClaims) -> VerifiedFederatedIdentity {
VerifiedFederatedIdentity::from_claims(
provider,
FederatedClaims {
sub: claims.sub,
email: claims.email,
username: claims.username,
groups: claims.groups,
raw: claims.raw,
},
)
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
use std::collections::HashMap;
#[test]
fn verified_identity_preserves_provider_and_claim_values() {
let raw = HashMap::from([
("iss".to_string(), json!("https://corp.example.test")),
("department".to_string(), json!("engineering")),
("roles".to_string(), json!(["reader", "admin"])),
]);
let identity = verified_identity(
FederatedProviderRef::new("corp".to_string()),
OidcClaims {
sub: " subject-123 ".to_string(),
email: " user@example.test ".to_string(),
username: " user ".to_string(),
groups: vec![
"source-ops".to_string(),
"source-developers".to_string(),
"source-ops".to_string(),
],
raw: raw.clone(),
},
);
assert_eq!(identity.provider().as_str(), "corp");
assert_eq!(identity.subject(), " subject-123 ");
assert_eq!(identity.email(), " user@example.test ");
assert_eq!(identity.username(), " user ");
assert_eq!(identity.source_groups(), ["source-ops", "source-developers", "source-ops"]);
assert_eq!(identity.attributes(), &raw);
}
}
crates/iam/src/federation/oidc/config.rs
-68
View File
@@ -1,68 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{
federation::{FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedRedirectPolicy},
oidc::{OidcConfigSnapshot, OidcSys},
};
pub(crate) fn authorization_rules(oidc: &OidcSys) -> FederatedAuthorizationRules {
FederatedAuthorizationRules::new(oidc.provider_configs().map(|config| {
FederatedAuthorizationRule::new(
config.id.clone(),
config.claim_name.clone(),
config.claim_prefix.clone(),
config.role_policy.clone(),
config.groups_claim.clone(),
config.roles_claim.clone(),
)
}))
}
/// Read-only access to the active standard OIDC configuration.
///
/// This interface stays separate from authentication and generic provider
/// views because its snapshots include OIDC-specific fields and secrets.
pub trait OidcConfigQuery: Send + Sync {
fn config_snapshot(&self) -> OidcConfigSnapshot;
}
pub(super) fn redirect_policy(oidc: &OidcSys, provider_id: &str) -> Option<FederatedRedirectPolicy> {
oidc.get_provider_config(provider_id).map(|config| FederatedRedirectPolicy {
redirect_uri: config.redirect_uri.clone(),
allow_request_origin: config.redirect_uri_dynamic,
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn redirect_policy_keeps_configured_uri_and_dynamic_setting() {
let mut config = crate::oidc::test_config("default");
config.redirect_uri = Some("https://console.example.test/callback".to_string());
config.redirect_uri_dynamic = false;
let oidc = crate::oidc::make_test_sys(vec![config]);
assert_eq!(
redirect_policy(&oidc, "default"),
Some(FederatedRedirectPolicy {
redirect_uri: Some("https://console.example.test/callback".to_string()),
allow_request_origin: false,
})
);
assert_eq!(redirect_policy(&oidc, "missing"), None);
}
}
crates/iam/src/federation/oidc/discovery.rs
-63
View File
@@ -1,63 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{federation::FederatedProviderView, oidc::OidcSys};
fn provider_view(provider: &crate::oidc::OidcProviderConfig) -> FederatedProviderView {
FederatedProviderView {
provider_id: provider.id.clone(),
display_name: provider.display_name.clone(),
}
}
pub(super) fn list_providers(oidc: &OidcSys) -> Vec<FederatedProviderView> {
oidc.provider_configs().map(provider_view).collect()
}
pub(super) fn list_visible_providers(oidc: &OidcSys) -> Vec<FederatedProviderView> {
oidc.provider_configs()
.filter(|provider| !provider.hide_from_ui)
.map(provider_view)
.collect()
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn provider_views_preserve_names_and_visibility() {
let mut visible = crate::oidc::test_config("default");
visible.display_name = "Company SSO".to_string();
let mut hidden = crate::oidc::test_config("workload");
hidden.hide_from_ui = true;
let oidc = crate::oidc::make_test_sys(vec![visible, hidden]);
let all = list_providers(&oidc);
assert_eq!(all.len(), 2);
assert!(
all.iter()
.any(|provider| { provider.provider_id == "default" && provider.display_name == "Company SSO" })
);
assert!(all.iter().any(|provider| provider.provider_id == "workload"));
assert_eq!(
list_visible_providers(&oidc),
vec![FederatedProviderView {
provider_id: "default".to_string(),
display_name: "Company SSO".to_string(),
}]
);
}
}
crates/iam/src/federation/oidc/flow.rs
-136
View File
@@ -1,136 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::config::OidcConfigQuery;
use super::{claims, config, discovery, http};
use crate::{
federation::{
CoreFederatedAuthorizationMapper, FederatedAuthorization, FederatedAuthorizationRules, FederatedCodeExchange,
FederatedIdentityProvider, FederatedIdentityService, FederatedProviderRef, FederatedProviderView,
FederatedRedirectPolicy, FederationError, OpaqueLogoutContinuation, Result, StandardOidcAuthentication,
VerifiedFederatedCodeExchange,
},
oidc::{OidcConfigSnapshot, OidcSys},
};
use std::sync::Arc;
pub struct StandardOidcAdapter {
oidc: Arc<OidcSys>,
authorization_rules: FederatedAuthorizationRules,
}
impl StandardOidcAdapter {
pub fn new(oidc: Arc<OidcSys>) -> Self {
let authorization_rules = config::authorization_rules(&oidc);
Self {
oidc,
authorization_rules,
}
}
pub fn authorization_rules(&self) -> FederatedAuthorizationRules {
self.authorization_rules.clone()
}
pub fn into_service(self: Arc<Self>, mapper: CoreFederatedAuthorizationMapper) -> FederatedIdentityService {
let authentication: Arc<dyn StandardOidcAuthentication> = self.clone();
let provider: Arc<dyn FederatedIdentityProvider> = self;
FederatedIdentityService::from_standard_oidc_parts(provider, authentication, mapper)
}
}
impl OidcConfigQuery for StandardOidcAdapter {
fn config_snapshot(&self) -> OidcConfigSnapshot {
self.oidc.config_snapshot()
}
}
#[async_trait::async_trait]
impl StandardOidcAuthentication for StandardOidcAdapter {
async fn exchange_identity(&self, state: &str, code: &str, redirect_uri: &str) -> Result<VerifiedFederatedCodeExchange> {
let (oidc_claims, provider_id, session, id_token) = http::exchange_code(&self.oidc, state, code, redirect_uri).await?;
let provider = FederatedProviderRef::new(provider_id);
Ok(VerifiedFederatedCodeExchange::new(
claims::verified_identity(provider.clone(), oidc_claims),
session.redirect_after,
OpaqueLogoutContinuation::new(provider, id_token),
))
}
async fn verify_identity(&self, jwt: &str) -> Result<crate::federation::VerifiedFederatedIdentity> {
let (oidc_claims, provider_id) = self
.oidc
.verify_web_identity_token(jwt)
.await
.map_err(FederationError::TokenVerification)?;
Ok(claims::verified_identity(FederatedProviderRef::new(provider_id), oidc_claims))
}
async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> Result<String> {
let (provider, id_token) = continuation.into_parts();
http::create_logout_token(&self.oidc, provider.as_str(), &id_token).await
}
}
#[async_trait::async_trait]
impl FederatedIdentityProvider for StandardOidcAdapter {
fn has_providers(&self) -> bool {
self.oidc.has_providers()
}
fn list_providers(&self) -> Vec<FederatedProviderView> {
discovery::list_providers(&self.oidc)
}
fn list_visible_providers(&self) -> Vec<FederatedProviderView> {
discovery::list_visible_providers(&self.oidc)
}
fn provider_config(&self, provider_id: &str) -> Option<&crate::oidc::OidcProviderConfig> {
self.oidc.get_provider_config(provider_id)
}
fn redirect_policy(&self, provider_id: &str) -> Option<FederatedRedirectPolicy> {
config::redirect_policy(&self.oidc, provider_id)
}
async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option<String>) -> Result<String> {
http::authorize_url(&self.oidc, provider_id, redirect_uri, redirect_after).await
}
async fn exchange_code(&self, state: &str, code: &str, redirect_uri: &str) -> Result<FederatedCodeExchange> {
let exchange = StandardOidcAuthentication::exchange_identity(self, state, code, redirect_uri).await?;
let (identity, redirect_after, continuation) = exchange.into_parts();
let authorization = CoreFederatedAuthorizationMapper::new(self.authorization_rules()).map(identity);
let (_, id_token) = continuation.into_parts();
Ok(FederatedCodeExchange {
authorization,
redirect_after,
id_token,
})
}
async fn verify_web_identity_token(&self, jwt: &str) -> Result<FederatedAuthorization> {
let identity = StandardOidcAuthentication::verify_identity(self, jwt).await?;
Ok(CoreFederatedAuthorizationMapper::new(self.authorization_rules()).map(identity))
}
async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result<String> {
http::create_logout_token(&self.oidc, provider_id, id_token).await
}
async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result<Option<String>> {
http::build_logout_url(&self.oidc, logout_token, post_logout_redirect_uri).await
}
}
crates/iam/src/federation/oidc/http.rs
-57
View File
@@ -1,57 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{
federation::{FederationError, Result},
oidc::{OidcClaims, OidcSys},
oidc_state::OidcAuthSession,
};
pub(super) async fn authorize_url(
oidc: &OidcSys,
provider_id: &str,
redirect_uri: &str,
redirect_after: Option<String>,
) -> Result<String> {
oidc.authorize_url(provider_id, redirect_uri, redirect_after)
.await
.map_err(FederationError::Authorization)
}
pub(super) async fn exchange_code(
oidc: &OidcSys,
state: &str,
code: &str,
redirect_uri: &str,
) -> Result<(OidcClaims, String, OidcAuthSession, String)> {
oidc.exchange_code(state, code, redirect_uri)
.await
.map_err(FederationError::CodeExchange)
}
pub(super) async fn create_logout_token(oidc: &OidcSys, provider_id: &str, id_token: &str) -> Result<String> {
oidc.create_logout_token(provider_id, id_token)
.await
.map_err(FederationError::Logout)
}
pub(super) async fn build_logout_url(
oidc: &OidcSys,
logout_token: &str,
post_logout_redirect_uri: &str,
) -> Result<Option<String>> {
oidc.build_logout_url(logout_token, post_logout_redirect_uri)
.await
.map_err(FederationError::Logout)
}
crates/iam/src/federation/oidc/mod.rs
-22
View File
@@ -1,22 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
mod claims;
mod config;
mod discovery;
mod flow;
mod http;
pub use config::OidcConfigQuery;
pub use flow::StandardOidcAdapter;
crates/iam/src/federation/provider.rs
+5 -26
View File
@@ -12,8 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use super::{FederatedAuthorization, FederatedCodeExchange, OpaqueLogoutContinuation, Result, VerifiedFederatedIdentity};
use crate::oidc::OidcProviderConfig;
use super::{OpaqueLogoutContinuation, Result, VerifiedFederatedIdentity};
use serde::{Deserialize, Serialize};
/// Provider data exposed to login discovery consumers.
@@ -56,41 +55,21 @@ impl VerifiedFederatedCodeExchange {
#[async_trait::async_trait]
pub(crate) trait StandardOidcAuthentication: Send + Sync {
async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option<String>) -> Result<String>;
async fn exchange_identity(&self, state: &str, code: &str, redirect_uri: &str) -> Result<VerifiedFederatedCodeExchange>;
async fn verify_identity(&self, jwt: &str) -> Result<VerifiedFederatedIdentity>;
async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> Result<String>;
async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result<Option<String>>;
}
#[async_trait::async_trait]
pub trait FederatedIdentityProvider: Send + Sync {
pub(crate) trait FederatedProviderQuery: Send + Sync {
fn has_providers(&self) -> bool;
fn list_providers(&self) -> Vec<FederatedProviderView>;
fn list_visible_providers(&self) -> Vec<FederatedProviderView>;
/// Compatibility accessor for existing provider implementations.
/// Login redirect consumers should use [`Self::redirect_policy`].
fn provider_config(&self, _provider_id: &str) -> Option<&OidcProviderConfig> {
None
}
fn redirect_policy(&self, provider_id: &str) -> Option<FederatedRedirectPolicy> {
self.provider_config(provider_id).map(|config| FederatedRedirectPolicy {
redirect_uri: config.redirect_uri.clone(),
allow_request_origin: config.redirect_uri_dynamic,
})
}
async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option<String>) -> Result<String>;
async fn exchange_code(&self, state: &str, code: &str, redirect_uri: &str) -> Result<FederatedCodeExchange>;
async fn verify_web_identity_token(&self, jwt: &str) -> Result<FederatedAuthorization>;
async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result<String>;
async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result<Option<String>>;
fn redirect_policy(&self, provider_id: &str) -> Option<FederatedRedirectPolicy>;
}
#[cfg(test)]
crates/iam/src/federation/registry.rs
-32
View File
@@ -1,32 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::FederatedIdentityProvider;
use std::sync::Arc;
/// Compatibility facade for integrations built against the original provider API.
/// Production standard OIDC startup composes the adapter, mapper, and service directly.
pub struct FederatedIdentityRegistry {
standard_oidc: Arc<dyn FederatedIdentityProvider>,
}
impl FederatedIdentityRegistry {
pub fn new(standard_oidc: Arc<dyn FederatedIdentityProvider>) -> Self {
Self { standard_oidc }
}
pub(crate) fn standard_oidc_arc(&self) -> Arc<dyn FederatedIdentityProvider> {
Arc::clone(&self.standard_oidc)
}
}
crates/iam/src/federation/transaction.rs
+106 -218
View File
@@ -13,11 +13,10 @@
// limitations under the License.
use super::{
CoreFederatedAuthorizationMapper, FederatedAuthorization, FederatedIdentityProvider, FederatedIdentityRegistry,
FederatedLoginSession, FederatedProviderView, FederatedRedirectPolicy, FederatedSession, FederatedSessionBinding,
FederatedSessionTransaction, FederationError, OpaqueLogoutContinuation, Result, StandardOidcAuthentication,
CoreFederatedAuthorizationMapper, FederatedAuthorization, FederatedLoginSession, FederatedProviderQuery,
FederatedProviderView, FederatedRedirectPolicy, FederatedSession, FederatedSessionBinding, FederatedSessionTransaction,
FederationError, OpaqueLogoutContinuation, Result, StandardOidcAuthentication,
};
use crate::oidc::OidcProviderConfig;
use std::sync::Arc;
const DEFAULT_OIDC_PROVIDER_ID: &str = "default";
@@ -32,11 +31,7 @@ fn sorted_provider_views(mut providers: Vec<FederatedProviderView>) -> Vec<Feder
}
pub struct FederatedIdentityService {
provider: Arc<dyn FederatedIdentityProvider>,
standard_oidc: Option<StandardOidcRuntime>,
}
struct StandardOidcRuntime {
provider_query: Arc<dyn FederatedProviderQuery>,
authentication: Arc<dyn StandardOidcAuthentication>,
mapper: CoreFederatedAuthorizationMapper,
}
@@ -49,91 +44,57 @@ struct RuntimeCodeExchange {
impl FederatedIdentityService {
async fn exchange_code(&self, state: &str, code: &str, redirect_uri: &str) -> Result<RuntimeCodeExchange> {
match &self.standard_oidc {
Some(runtime) => {
let exchange = runtime.authentication.exchange_identity(state, code, redirect_uri).await?;
let (identity, redirect_after, logout_continuation) = exchange.into_parts();
Ok(RuntimeCodeExchange {
authorization: runtime.mapper.map(identity),
redirect_after,
logout_continuation,
})
}
None => {
let exchange = self.provider.exchange_code(state, code, redirect_uri).await?;
let (authorization, redirect_after, logout_continuation) = exchange.into_parts();
Ok(RuntimeCodeExchange {
authorization,
redirect_after,
logout_continuation,
})
}
}
let exchange = self.authentication.exchange_identity(state, code, redirect_uri).await?;
let (identity, redirect_after, logout_continuation) = exchange.into_parts();
Ok(RuntimeCodeExchange {
authorization: self.mapper.map(identity),
redirect_after,
logout_continuation,
})
}
async fn verify_identity(&self, jwt: &str) -> Result<FederatedAuthorization> {
match &self.standard_oidc {
Some(runtime) => Ok(runtime.mapper.map(runtime.authentication.verify_identity(jwt).await?)),
None => self.provider.verify_web_identity_token(jwt).await,
}
Ok(self.mapper.map(self.authentication.verify_identity(jwt).await?))
}
async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> Result<String> {
match &self.standard_oidc {
Some(runtime) => runtime.authentication.create_logout_token(continuation).await,
None => {
let (provider, id_token) = continuation.into_parts();
self.provider.create_logout_token(provider.as_str(), &id_token).await
}
}
self.authentication.create_logout_token(continuation).await
}
/// Build the compatibility service around the published provider interface.
pub fn new(registry: FederatedIdentityRegistry) -> Self {
Self {
provider: registry.standard_oidc_arc(),
standard_oidc: None,
}
}
pub(super) fn from_standard_oidc_parts(
provider: Arc<dyn FederatedIdentityProvider>,
pub(crate) fn from_standard_oidc_parts(
provider_query: Arc<dyn FederatedProviderQuery>,
authentication: Arc<dyn StandardOidcAuthentication>,
mapper: CoreFederatedAuthorizationMapper,
) -> Self {
Self {
provider,
standard_oidc: Some(StandardOidcRuntime { authentication, mapper }),
provider_query,
authentication,
mapper,
}
}
pub fn has_providers(&self) -> bool {
self.provider.has_providers()
self.provider_query.has_providers()
}
pub fn list_providers(&self) -> Vec<FederatedProviderView> {
let providers = self.provider.list_providers();
let providers = self.provider_query.list_providers();
sorted_provider_views(providers)
}
pub fn list_visible_providers(&self) -> Vec<FederatedProviderView> {
let providers = self.provider.list_visible_providers();
let providers = self.provider_query.list_visible_providers();
sorted_provider_views(providers)
}
pub fn redirect_policy(&self, provider_id: &str) -> Option<FederatedRedirectPolicy> {
self.provider.redirect_policy(provider_id)
}
/// Compatibility accessor for existing federation consumers.
/// Login redirects use [`Self::redirect_policy`], while OIDC-specific
/// configuration consumers use the OIDC configuration query.
pub fn get_provider_config(&self, provider_id: &str) -> Option<&OidcProviderConfig> {
self.provider.provider_config(provider_id)
self.provider_query.redirect_policy(provider_id)
}
pub async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option<String>) -> Result<String> {
self.provider.authorize_url(provider_id, redirect_uri, redirect_after).await
self.authentication
.authorize_url(provider_id, redirect_uri, redirect_after)
.await
}
pub async fn complete_authorization_code(
@@ -205,7 +166,9 @@ impl FederatedIdentityService {
}
pub async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result<Option<String>> {
self.provider.build_logout_url(logout_token, post_logout_redirect_uri).await
self.authentication
.build_logout_url(logout_token, post_logout_redirect_uri)
.await
}
}
@@ -213,9 +176,8 @@ impl FederatedIdentityService {
mod tests {
use super::*;
use crate::federation::{
FederatedAuthorization, FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedClaims, FederatedCodeExchange,
FederatedIdentityProvider, FederatedProviderRef, FederatedSessionBindingError, VerifiedFederatedCodeExchange,
VerifiedFederatedIdentity,
FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedClaims, FederatedProviderQuery, FederatedProviderRef,
FederatedSessionBindingError, VerifiedFederatedCodeExchange, VerifiedFederatedIdentity,
};
use rustfs_credentials::Credentials;
use std::sync::{Arc, Mutex};
@@ -235,6 +197,8 @@ mod tests {
web_provider_id: &'static str,
failure: ProviderFailure,
events: Arc<Mutex<Vec<&'static str>>>,
authorize_calls: Mutex<Vec<(String, String, Option<String>)>>,
logout_url_calls: Mutex<Vec<(String, String)>>,
expected_logout: (&'static str, &'static str),
listed_provider_ids: Vec<&'static str>,
visible_provider_ids: Vec<&'static str>,
@@ -249,6 +213,8 @@ mod tests {
web_provider_id: DEFAULT_OIDC_PROVIDER_ID,
failure: ProviderFailure::None,
events,
authorize_calls: Mutex::new(Vec::new()),
logout_url_calls: Mutex::new(Vec::new()),
expected_logout: (DEFAULT_OIDC_PROVIDER_ID, "id-token"),
listed_provider_ids: Vec::new(),
visible_provider_ids: Vec::new(),
@@ -259,32 +225,6 @@ mod tests {
self.events.lock().expect("event log should not be poisoned").push(event);
}
fn authorization(&self, provider_id: &str) -> FederatedAuthorization {
FederatedAuthorization {
provider_id: provider_id.to_string(),
claims: FederatedClaims {
sub: "subject".to_string(),
email: String::new(),
username: "user".to_string(),
groups: vec!["source-group".to_string()],
raw: Default::default(),
},
policies: if self.with_policy {
vec!["readwrite".to_string()]
} else {
Vec::new()
},
group_claim_policies: Vec::new(),
groups: if self.with_group {
vec!["developers".to_string()]
} else {
Vec::new()
},
roles_claim_key: None,
roles: Vec::new(),
}
}
fn identity(&self, provider_id: &str) -> VerifiedFederatedIdentity {
VerifiedFederatedIdentity::from_claims(
FederatedProviderRef::new(provider_id.to_string()),
@@ -313,8 +253,7 @@ mod tests {
.collect()
}
#[async_trait::async_trait]
impl FederatedIdentityProvider for TestProvider {
impl FederatedProviderQuery for TestProvider {
fn has_providers(&self) -> bool {
true
}
@@ -333,52 +272,18 @@ mod tests {
allow_request_origin: true,
})
}
async fn authorize_url(
&self,
_provider_id: &str,
_redirect_uri: &str,
_redirect_after: Option<String>,
) -> Result<String> {
Ok("https://identity.example/authorize".to_string())
}
async fn exchange_code(&self, _state: &str, _code: &str, _redirect_uri: &str) -> Result<FederatedCodeExchange> {
self.record("exchange");
if self.failure == ProviderFailure::Exchange {
return Err(FederationError::CodeExchange("exchange failed".to_string()));
}
Ok(FederatedCodeExchange {
authorization: self.authorization(self.browser_provider_id),
redirect_after: Some("/browser".to_string()),
id_token: "id-token".to_string(),
})
}
async fn verify_web_identity_token(&self, _jwt: &str) -> Result<FederatedAuthorization> {
self.record("verify");
if self.failure == ProviderFailure::Verification {
return Err(FederationError::TokenVerification("verification failed".to_string()));
}
Ok(self.authorization(self.web_provider_id))
}
async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result<String> {
self.record("logout");
assert_eq!((provider_id, id_token), self.expected_logout);
if self.failure == ProviderFailure::Logout {
return Err(FederationError::Logout("logout failed".to_string()));
}
Ok("logout-token".to_string())
}
async fn build_logout_url(&self, _logout_token: &str, _post_logout_redirect_uri: &str) -> Result<Option<String>> {
Ok(Some("https://identity.example/logout".to_string()))
}
}
#[async_trait::async_trait]
impl StandardOidcAuthentication for TestProvider {
async fn authorize_url(&self, provider_id: &str, redirect_uri: &str, redirect_after: Option<String>) -> Result<String> {
self.authorize_calls
.lock()
.expect("authorize call log should not be poisoned")
.push((provider_id.to_string(), redirect_uri.to_string(), redirect_after));
Ok("https://identity.example/authorize".to_string())
}
async fn exchange_identity(
&self,
_state: &str,
@@ -414,6 +319,14 @@ mod tests {
}
Ok("logout-token".to_string())
}
async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> Result<Option<String>> {
self.logout_url_calls
.lock()
.expect("logout URL call log should not be poisoned")
.push((logout_token.to_string(), post_logout_redirect_uri.to_string()));
Ok(Some("https://identity.example/logout".to_string()))
}
}
fn standard_service(provider: Arc<TestProvider>) -> FederatedIdentityService {
@@ -432,14 +345,12 @@ mod tests {
)
}));
let authentication: Arc<dyn StandardOidcAuthentication> = provider.clone();
let provider: Arc<dyn FederatedIdentityProvider> = provider;
FederatedIdentityService {
provider,
standard_oidc: Some(StandardOidcRuntime {
authentication,
mapper: CoreFederatedAuthorizationMapper::new(rules),
}),
}
let provider_query: Arc<dyn FederatedProviderQuery> = provider;
FederatedIdentityService::from_standard_oidc_parts(
provider_query,
authentication,
CoreFederatedAuthorizationMapper::new(rules),
)
}
struct RecordingBinding {
@@ -492,7 +403,7 @@ mod tests {
let mut provider = TestProvider::new(events);
provider.listed_provider_ids = vec!["zeta", "hidden", DEFAULT_OIDC_PROVIDER_ID, "alpha"];
provider.visible_provider_ids = vec!["zeta", DEFAULT_OIDC_PROVIDER_ID, "alpha"];
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(Arc::new(provider)));
let service = standard_service(Arc::new(provider));
assert_eq!(
service
@@ -519,6 +430,48 @@ mod tests {
);
}
#[tokio::test]
async fn authorization_and_logout_urls_preserve_service_arguments() {
let provider = Arc::new(TestProvider::new(Arc::new(Mutex::new(Vec::new()))));
let service = standard_service(provider.clone());
let authorize_url = service
.authorize_url(
"corp",
"https://console.example.com/oauth/callback",
Some("/browser/path?tab=objects".to_string()),
)
.await
.expect("authorization URL should be returned");
let logout_url = service
.build_logout_url("opaque-logout-token", "https://console.example.com/signed-out")
.await
.expect("logout URL should be returned");
assert_eq!(authorize_url, "https://identity.example/authorize");
assert_eq!(logout_url.as_deref(), Some("https://identity.example/logout"));
assert_eq!(
provider
.authorize_calls
.lock()
.expect("authorize call log should not be poisoned")
.as_slice(),
[(
"corp".to_string(),
"https://console.example.com/oauth/callback".to_string(),
Some("/browser/path?tab=objects".to_string()),
)]
);
assert_eq!(
provider
.logout_url_calls
.lock()
.expect("logout URL call log should not be poisoned")
.as_slice(),
[("opaque-logout-token".to_string(), "https://console.example.com/signed-out".to_string(),)]
);
}
#[tokio::test]
async fn callback_and_web_identity_preserve_provider_and_transaction_boundaries() {
let events = Arc::new(Mutex::new(Vec::new()));
@@ -526,9 +479,8 @@ mod tests {
provider.browser_provider_id = "corp";
provider.web_provider_id = "partner";
provider.expected_logout = ("corp", "id-token");
let provider = Arc::new(provider);
let binding = Arc::new(RecordingBinding::new(events.clone()));
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(provider));
let service = standard_service(Arc::new(provider));
let login = service
.complete_authorization_code("state", "code", "https://console.example/callback", 3600, binding.as_ref())
@@ -604,76 +556,13 @@ mod tests {
}
#[tokio::test]
async fn standard_runtime_checks_authorization_before_binding() {
async fn web_identity_without_policy_or_group_is_not_bound() {
let events = Arc::new(Mutex::new(Vec::new()));
let mut provider = TestProvider::new(events.clone());
provider.with_policy = false;
let binding = Arc::new(RecordingBinding::new(events.clone()));
let service = standard_service(Arc::new(provider));
let error = service
.assume_role_with_web_identity("jwt", 3600, None, binding.as_ref())
.await
.expect_err("standard web identity requires mapped authorization");
assert!(matches!(error, FederationError::NoAuthorizationContext));
assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify"]);
}
#[tokio::test]
async fn standard_runtime_preserves_browser_failure_order() {
for (provider_failure, binding_failure, expected_events) in [
(ProviderFailure::Exchange, false, vec!["exchange"]),
(ProviderFailure::None, true, vec!["exchange", "bind"]),
(ProviderFailure::Logout, false, vec!["exchange", "bind", "logout"]),
] {
let events = Arc::new(Mutex::new(Vec::new()));
let mut provider = TestProvider::new(events.clone());
provider.failure = provider_failure;
let mut binding = RecordingBinding::new(events.clone());
binding.fail = binding_failure;
let service = standard_service(Arc::new(provider));
service
.complete_authorization_code("state", "code", "https://console.example/callback", 3600, &binding)
.await
.expect_err("the configured standard callback failure should be returned");
assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), expected_events);
}
}
#[tokio::test]
async fn standard_runtime_preserves_web_identity_failure_order() {
for (provider_failure, binding_failure, expected_events) in [
(ProviderFailure::Verification, false, vec!["verify"]),
(ProviderFailure::None, true, vec!["verify", "bind"]),
] {
let events = Arc::new(Mutex::new(Vec::new()));
let mut provider = TestProvider::new(events.clone());
provider.failure = provider_failure;
let mut binding = RecordingBinding::new(events.clone());
binding.fail = binding_failure;
let service = standard_service(Arc::new(provider));
service
.assume_role_with_web_identity("jwt", 3600, None, &binding)
.await
.expect_err("the configured standard web identity failure should be returned");
assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), expected_events);
}
}
#[tokio::test]
async fn web_identity_without_policy_or_group_is_not_bound() {
let events = Arc::new(Mutex::new(Vec::new()));
let mut provider = TestProvider::new(events.clone());
provider.with_policy = false;
let provider = Arc::new(provider);
let binding = Arc::new(RecordingBinding::new(events.clone()));
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(provider));
let error = service
.assume_role_with_web_identity("jwt", 3600, None, binding.as_ref())
.await
@@ -689,16 +578,15 @@ mod tests {
let mut provider = TestProvider::new(events.clone());
provider.with_policy = false;
provider.with_group = true;
let provider = Arc::new(provider);
let binding = Arc::new(RecordingBinding::new(events.clone()));
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(provider));
let service = standard_service(Arc::new(provider));
let session = service
.assume_role_with_web_identity("jwt", 3600, None, binding.as_ref())
.await
.expect("a mapped group is an authorization context");
assert!(session.authorization.policies.is_empty());
assert_eq!(session.authorization.policies, ["developers"]);
assert_eq!(session.authorization.groups, ["developers"]);
assert_eq!(events.lock().expect("event log should not be poisoned").as_slice(), ["verify", "bind"]);
}
@@ -716,7 +604,7 @@ mod tests {
let mut binding = RecordingBinding::new(events.clone());
binding.fail = binding_failure;
let binding = Arc::new(binding);
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(Arc::new(provider)));
let service = standard_service(Arc::new(provider));
let error = service
.complete_authorization_code("state", "code", "https://console.example/callback", 3600, binding.as_ref())
@@ -755,7 +643,7 @@ mod tests {
let mut binding = RecordingBinding::new(events.clone());
binding.fail = binding_failure;
let binding = Arc::new(binding);
let service = FederatedIdentityService::new(FederatedIdentityRegistry::new(Arc::new(provider)));
let service = standard_service(Arc::new(provider));
let error = service
.assume_role_with_web_identity("jwt", 3600, None, binding.as_ref())
crates/iam/src/lib.rs
+119 -49
View File
@@ -34,7 +34,6 @@ pub mod keyring;
pub mod manager;
pub mod mfa;
pub mod oidc;
pub mod oidc_state;
mod root_credentials;
mod runtime_sources;
mod server_config;
@@ -194,7 +193,6 @@ pub(crate) async fn notify_iam_load_policy_mapping(
}
static IAM_SYS: OnceLock<Arc<IamSys<ObjectStore>>> = OnceLock::new();
static OIDC_SYS: OnceLock<Arc<OidcSys>> = OnceLock::new();
/// Build an IAM system bound to the given store without touching the process
/// singleton (backlog#1052 S3): a per-server context can own the returned
@@ -284,36 +282,11 @@ pub fn get_global_iam_sys() -> Option<Arc<IamSys<ObjectStore>>> {
IAM_SYS.get().cloned()
}
/// Initialize the global OIDC system. Non-fatal if no OIDC providers are configured.
pub async fn init_oidc_sys() -> Result<()> {
init_oidc_sys_with_extra_root_ca(None).await
}
/// Initialize the global OIDC system with an additional outbound root CA bundle.
pub async fn init_oidc_sys_with_extra_root_ca(root_ca_pem: Option<&[u8]>) -> Result<()> {
init_oidc_sys_with_extra_root_ca_provider_inner(None, root_ca_pem).await
}
/// Initialize the global OIDC system with a reload-aware outbound root CA provider.
pub async fn init_oidc_sys_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<()> {
init_oidc_sys_with_extra_root_ca_provider_inner(Some(extra_root_ca_provider), None).await
}
async fn init_oidc_sys_with_extra_root_ca_provider_inner(
extra_root_ca_provider: Option<OidcExtraRootCaProvider>,
root_ca_pem: Option<&[u8]>,
) -> Result<()> {
if OIDC_SYS.get().is_some() {
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "already_initialized",
"OIDC runtime already initialized"
);
return Ok(());
}
/// Build the OIDC system with reload-aware outbound root CA material.
///
/// Invalid OIDC setup remains non-fatal: the returned system has no providers,
/// matching the server startup behavior when OIDC cannot be initialized.
pub async fn build_oidc_sys_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<Arc<OidcSys>> {
debug!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
@@ -322,10 +295,10 @@ async fn init_oidc_sys_with_extra_root_ca_provider_inner(
"OIDC runtime starting"
);
let oidc_sys_result = match extra_root_ca_provider {
Some(provider) => OidcSys::new_with_extra_root_ca_provider(provider).await,
None => OidcSys::new_with_extra_root_ca(root_ca_pem).await,
};
finish_oidc_sys_build(OidcSys::new_with_extra_root_ca_provider(extra_root_ca_provider).await)
}
fn finish_oidc_sys_build(oidc_sys_result: std::result::Result<OidcSys, String>) -> Result<Arc<OidcSys>> {
let oidc_sys = match oidc_sys_result {
Ok(sys) => {
if sys.has_providers() {
@@ -361,20 +334,117 @@ async fn init_oidc_sys_with_extra_root_ca_provider_inner(
}
};
if OIDC_SYS.set(Arc::new(oidc_sys)).is_err() {
warn!(
event = EVENT_OIDC_STATE,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
state = "singleton_set_race",
"OIDC runtime singleton set raced"
);
Ok(Arc::new(oidc_sys))
}
#[cfg(test)]
mod tests {
use super::*;
use std::{
io::{self, Write},
sync::{
Mutex,
atomic::{AtomicUsize, Ordering},
},
};
use tracing_subscriber::{Registry, fmt::MakeWriter, layer::SubscriberExt};
#[derive(Clone, Default)]
struct CapturedLogs(Arc<Mutex<Vec<u8>>>);
struct CapturedLogWriter(Arc<Mutex<Vec<u8>>>);
impl Write for CapturedLogWriter {
fn write(&mut self, buf: &[u8]) -> io::Result<usize> {
self.0
.lock()
.expect("captured log lock should be available")
.extend_from_slice(buf);
Ok(buf.len())
}
fn flush(&mut self) -> io::Result<()> {
Ok(())
}
}
Ok(())
}
impl<'writer> MakeWriter<'writer> for CapturedLogs {
type Writer = CapturedLogWriter;
/// Get the global OIDC system.
pub fn get_oidc() -> Option<Arc<OidcSys>> {
OIDC_SYS.get().cloned()
fn make_writer(&'writer self) -> Self::Writer {
CapturedLogWriter(Arc::clone(&self.0))
}
}
fn capture_oidc_build(result: std::result::Result<OidcSys, String>) -> (Arc<OidcSys>, String) {
let logs = CapturedLogs::default();
let captured = Arc::clone(&logs.0);
let subscriber = Registry::default().with(
tracing_subscriber::fmt::layer()
.without_time()
.with_target(false)
.with_level(true)
.with_ansi(false)
.with_writer(logs),
);
let oidc = tracing::subscriber::with_default(subscriber, || {
finish_oidc_sys_build(result).expect("OIDC build completion should succeed")
});
let output = String::from_utf8(captured.lock().expect("captured log lock should be available").clone())
.expect("captured logs should be UTF-8");
(oidc, output)
}
#[test]
fn oidc_build_reports_empty_configuration() {
let (oidc, output) = capture_oidc_build(OidcSys::empty());
assert!(!oidc.has_providers());
assert!(output.contains(EVENT_OIDC_STATE), "{output}");
assert!(output.contains("state=\"empty\""), "{output}");
}
#[test]
fn oidc_build_reports_ready_configuration() {
let oidc = crate::oidc::make_test_sys(vec![crate::oidc::test_config("default")]);
let (oidc, output) = capture_oidc_build(Ok(oidc));
assert!(oidc.has_providers());
assert_eq!(oidc.config_snapshot().providers().len(), 1);
assert!(output.contains(EVENT_OIDC_STATE), "{output}");
assert!(output.contains("state=\"ready\""), "{output}");
assert!(output.contains("provider_count=1"), "{output}");
}
#[test]
fn oidc_build_reports_error_and_returns_empty_runtime() {
let (oidc, output) = capture_oidc_build(Err("configured OIDC root CA is invalid".to_string()));
assert!(!oidc.has_providers());
assert!(output.contains("WARN"), "{output}");
assert!(output.contains(EVENT_OIDC_STATE), "{output}");
assert!(output.contains("state=\"init_failed_non_fatal\""), "{output}");
assert!(output.contains("configured OIDC root CA is invalid"), "{output}");
}
#[tokio::test]
async fn oidc_builder_loads_root_ca_provider_and_keeps_failure_non_fatal() {
let load_count = Arc::new(AtomicUsize::new(0));
let observed_load_count = Arc::clone(&load_count);
let provider = OidcExtraRootCaProvider::new(move || {
let observed_load_count = Arc::clone(&observed_load_count);
async move {
observed_load_count.fetch_add(1, Ordering::SeqCst);
Err("test root CA load failure".to_string())
}
});
let oidc = build_oidc_sys_with_extra_root_ca_provider(provider)
.await
.expect("root CA load failure should produce an empty OIDC runtime");
assert_eq!(load_count.load(Ordering::SeqCst), 1);
assert!(!oidc.has_providers());
}
}
crates/iam/src/oidc.rs
-4708
View File
File diff suppressed because it is too large Load Diff
crates/iam/src/oidc/config.rs
+867
View File
@@ -0,0 +1,867 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use base64_simd::URL_SAFE_NO_PAD;
use rustfs_config::oidc::*;
use rustfs_config::server_config::{Config as ServerConfig, KVS};
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, EnableState};
use rustfs_utils::egress::{OutboundUrlError, validate_outbound_url};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use std::fmt;
use url::Url;
pub(super) const REDACTED_SECRET: &str = "***redacted***";
fn redacted_optional_secret(value: Option<&str>) -> &'static str {
value.filter(|secret| !secret.is_empty()).map_or("", |_| REDACTED_SECRET)
}
/// Parsed configuration for a single OIDC provider.
#[derive(Clone, PartialEq, Eq)]
pub struct OidcProviderConfig {
pub id: String,
pub enabled: bool,
pub config_url: String,
pub issuer: Option<String>,
pub client_id: String,
pub client_secret: Option<String>,
pub scopes: Vec<String>,
pub other_audiences: Vec<String>,
pub redirect_uri: Option<String>,
pub redirect_uri_dynamic: bool,
pub claim_name: String,
pub claim_prefix: String,
pub role_policy: String,
pub display_name: String,
pub groups_claim: String,
pub roles_claim: String,
pub email_claim: String,
pub username_claim: String,
pub hide_from_ui: bool,
}
#[derive(Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct OidcProviderConfigInput {
pub enabled: bool,
pub display_name: String,
pub config_url: String,
pub issuer: Option<String>,
pub client_id: String,
pub client_secret: Option<String>,
pub scopes: Vec<String>,
pub other_audiences: Vec<String>,
pub redirect_uri: Option<String>,
pub redirect_uri_dynamic: bool,
pub claim_name: String,
pub claim_prefix: String,
pub role_policy: String,
pub groups_claim: String,
pub roles_claim: String,
pub email_claim: String,
pub username_claim: String,
pub hide_from_ui: bool,
}
impl Default for OidcProviderConfigInput {
fn default() -> Self {
Self {
enabled: true,
display_name: String::new(),
config_url: String::new(),
issuer: None,
client_id: String::new(),
client_secret: None,
scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(),
other_audiences: Vec::new(),
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: false,
}
}
}
#[derive(Deserialize)]
#[serde(default, deny_unknown_fields)]
pub struct OidcProviderValidationInput {
pub provider_id: String,
pub enabled: bool,
pub display_name: String,
pub config_url: String,
pub issuer: Option<String>,
pub client_id: String,
pub client_secret: Option<String>,
pub scopes: Vec<String>,
pub other_audiences: Vec<String>,
pub redirect_uri: Option<String>,
pub redirect_uri_dynamic: bool,
pub claim_name: String,
pub claim_prefix: String,
pub role_policy: String,
pub groups_claim: String,
pub roles_claim: String,
pub email_claim: String,
pub username_claim: String,
pub hide_from_ui: bool,
}
impl Default for OidcProviderValidationInput {
fn default() -> Self {
let input = OidcProviderConfigInput::default();
Self {
provider_id: "default".to_string(),
enabled: input.enabled,
display_name: input.display_name,
config_url: input.config_url,
issuer: input.issuer,
client_id: input.client_id,
client_secret: input.client_secret,
scopes: input.scopes,
other_audiences: input.other_audiences,
redirect_uri: input.redirect_uri,
redirect_uri_dynamic: input.redirect_uri_dynamic,
claim_name: input.claim_name,
claim_prefix: input.claim_prefix,
role_policy: input.role_policy,
groups_claim: input.groups_claim,
roles_claim: input.roles_claim,
email_claim: input.email_claim,
username_claim: input.username_claim,
hide_from_ui: input.hide_from_ui,
}
}
}
impl From<OidcProviderValidationInput> for (String, OidcProviderConfigInput) {
fn from(input: OidcProviderValidationInput) -> Self {
let provider_id = if input.provider_id.trim().is_empty() {
"default".to_string()
} else {
input.provider_id.trim().to_string()
};
let config = OidcProviderConfigInput {
enabled: input.enabled,
display_name: input.display_name,
config_url: input.config_url,
issuer: input.issuer,
client_id: input.client_id,
client_secret: input.client_secret,
scopes: input.scopes,
other_audiences: input.other_audiences,
redirect_uri: input.redirect_uri,
redirect_uri_dynamic: input.redirect_uri_dynamic,
claim_name: input.claim_name,
claim_prefix: input.claim_prefix,
role_policy: input.role_policy,
groups_claim: input.groups_claim,
roles_claim: input.roles_claim,
email_claim: input.email_claim,
username_claim: input.username_claim,
hide_from_ui: input.hide_from_ui,
};
(provider_id, config)
}
}
#[derive(Debug, thiserror::Error)]
pub enum OidcConfigError {
#[error("invalid provider_id")]
InvalidProviderId,
#[error("provider is managed by environment variables")]
EnvironmentManaged,
#[error("provider not found")]
ProviderNotFound,
#[error("config_url is required")]
ConfigUrlRequired,
#[error("client_id is required")]
ClientIdRequired,
#[error("redirect_uri is required when redirect_uri_dynamic is off")]
RedirectUriRequired,
#[error("scopes must include openid")]
OpenidScopeRequired,
#[error("{0} must be an absolute http/https URL")]
InvalidAbsoluteUrl(&'static str),
#[error("{field} is not allowed: {source}")]
ForbiddenOutbound {
field: &'static str,
#[source]
source: OutboundUrlError,
},
}
impl fmt::Debug for OidcProviderConfig {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("OidcProviderConfig")
.field("id", &self.id)
.field("enabled", &self.enabled)
.field("config_url", &self.config_url)
.field("issuer", &self.issuer)
.field("client_id", &self.client_id)
.field("client_secret", &redacted_optional_secret(self.client_secret.as_deref()))
.field("scopes", &self.scopes)
.field("other_audiences", &self.other_audiences)
.field("redirect_uri", &self.redirect_uri)
.field("redirect_uri_dynamic", &self.redirect_uri_dynamic)
.field("claim_name", &self.claim_name)
.field("claim_prefix", &self.claim_prefix)
.field("role_policy", &self.role_policy)
.field("display_name", &self.display_name)
.field("groups_claim", &self.groups_claim)
.field("roles_claim", &self.roles_claim)
.field("email_claim", &self.email_claim)
.field("username_claim", &self.username_claim)
.field("hide_from_ui", &self.hide_from_ui)
.finish()
}
}
#[derive(Debug, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum OidcProviderConfigSource {
Env,
Persisted,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct SourcedOidcProviderConfig {
pub config: OidcProviderConfig,
pub source: OidcProviderConfigSource,
}
/// Immutable OIDC configuration entries returned by configuration queries.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OidcConfigSnapshot {
providers: Vec<SourcedOidcProviderConfig>,
}
impl OidcConfigSnapshot {
pub fn new(mut providers: Vec<SourcedOidcProviderConfig>) -> Self {
providers.sort_by(|left, right| {
(left.config.id != "default")
.cmp(&(right.config.id != "default"))
.then_with(|| left.config.id.cmp(&right.config.id))
});
Self { providers }
}
pub fn providers(&self) -> &[SourcedOidcProviderConfig] {
&self.providers
}
pub fn into_providers(self) -> Vec<SourcedOidcProviderConfig> {
self.providers
}
}
#[derive(Debug, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct OidcProviderValidationResult {
pub issuer: String,
pub authorization_endpoint: Option<String>,
pub token_endpoint: Option<String>,
}
/// Compatibility name for the provider summary returned by OIDC list APIs.
pub(crate) type OidcProviderSummary = crate::federation::FederatedProviderView;
/// OIDC fields required to compare identity providers across replicated sites.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OidcSiteReplicationProvider {
pub provider_id: String,
pub claim_name: String,
pub role_policy: String,
pub client_id: String,
pub hashed_client_secret: String,
}
/// Active OIDC settings exposed to site replication without raw credentials.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct OidcSiteReplicationSnapshot {
providers: Vec<OidcSiteReplicationProvider>,
}
impl OidcSiteReplicationSnapshot {
pub fn from_config_snapshot(snapshot: &OidcConfigSnapshot) -> Self {
let providers = snapshot
.providers()
.iter()
.map(|provider| {
let config = &provider.config;
OidcSiteReplicationProvider {
provider_id: config.id.clone(),
claim_name: config.claim_name.clone(),
role_policy: config.role_policy.clone(),
client_id: config.client_id.clone(),
hashed_client_secret: hash_client_secret(config.client_secret.as_deref()),
}
})
.collect();
Self { providers }
}
pub fn providers(&self) -> &[OidcSiteReplicationProvider] {
&self.providers
}
}
fn hash_client_secret(secret: Option<&str>) -> String {
let Some(secret) = secret.filter(|secret| !secret.is_empty()) else {
return String::new();
};
let mut hasher = Sha256::new();
hasher.update(secret.as_bytes());
URL_SAFE_NO_PAD.encode_to_string(hasher.finalize())
}
/// Read-only access to the active OIDC settings needed by site replication.
pub trait OidcConfigQuery: Send + Sync {
fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot;
}
pub fn validate_mutable_provider_id(provider_id: &str) -> Result<(), OidcConfigError> {
if !is_valid_provider_id(provider_id) {
return Err(OidcConfigError::InvalidProviderId);
}
if load_oidc_provider_configs_from_env()
.iter()
.any(|config| config.id == provider_id)
{
return Err(OidcConfigError::EnvironmentManaged);
}
Ok(())
}
fn is_valid_provider_id(id: &str) -> bool {
!id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
}
fn provider_instance_key(provider_id: &str) -> String {
if provider_id == "default" {
DEFAULT_DELIMITER.to_string()
} else {
provider_id.to_string()
}
}
fn normalize_optional(value: Option<String>) -> Option<String> {
value.map(|value| value.trim().to_string()).filter(|value| !value.is_empty())
}
fn or_default(value: &str, default: &str) -> String {
if value.trim().is_empty() {
default.to_string()
} else {
value.trim().to_string()
}
}
fn validate_absolute_http_url(value: &str, field: &'static str, check_outbound: bool) -> Result<(), OidcConfigError> {
let parsed = Url::parse(value).map_err(|_| OidcConfigError::InvalidAbsoluteUrl(field))?;
if !matches!(parsed.scheme(), "http" | "https") || parsed.host_str().is_none() {
return Err(OidcConfigError::InvalidAbsoluteUrl(field));
}
if check_outbound {
validate_outbound_url(&parsed).map_err(|source| OidcConfigError::ForbiddenOutbound { field, source })?;
}
Ok(())
}
fn normalize_provider_config(mut config: OidcProviderConfig) -> OidcProviderConfig {
config.config_url = config.config_url.trim().to_string();
config.issuer = normalize_optional(config.issuer);
config.client_id = config.client_id.trim().to_string();
config.scopes = config
.scopes
.iter()
.map(|scope| scope.trim().to_string())
.filter(|scope| !scope.is_empty())
.collect();
config.redirect_uri = normalize_optional(config.redirect_uri);
config.claim_name = or_default(&config.claim_name, OIDC_DEFAULT_CLAIM_NAME);
config.claim_prefix = config.claim_prefix.trim().to_string();
config.role_policy = config.role_policy.trim().to_string();
config.display_name = or_default(&config.display_name, &config.id);
config.groups_claim = or_default(&config.groups_claim, OIDC_DEFAULT_GROUPS_CLAIM);
config.roles_claim = or_default(&config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM);
config.email_claim = or_default(&config.email_claim, OIDC_DEFAULT_EMAIL_CLAIM);
config.username_claim = or_default(&config.username_claim, OIDC_DEFAULT_USERNAME_CLAIM);
config
}
fn validate_provider_config_fields(config: &OidcProviderConfig) -> Result<(), OidcConfigError> {
if !is_valid_provider_id(&config.id) {
return Err(OidcConfigError::InvalidProviderId);
}
if config.config_url.trim().is_empty() {
return Err(OidcConfigError::ConfigUrlRequired);
}
validate_absolute_http_url(&config.config_url, "config_url", true)?;
if let Some(issuer) = config.issuer.as_deref() {
validate_absolute_http_url(issuer, "issuer", false)?;
}
if config.client_id.trim().is_empty() {
return Err(OidcConfigError::ClientIdRequired);
}
if !config.redirect_uri_dynamic {
let redirect_uri = config.redirect_uri.as_deref().ok_or(OidcConfigError::RedirectUriRequired)?;
validate_absolute_http_url(redirect_uri, "redirect_uri", true)?;
} else if let Some(redirect_uri) = config.redirect_uri.as_deref() {
validate_absolute_http_url(redirect_uri, "redirect_uri", true)?;
}
if !config.scopes.iter().any(|scope| scope == "openid") {
return Err(OidcConfigError::OpenidScopeRequired);
}
Ok(())
}
fn build_provider_config(
provider_id: &str,
input: OidcProviderConfigInput,
existing_secret: Option<String>,
) -> Result<OidcProviderConfig, OidcConfigError> {
let client_secret = match input.client_secret {
Some(value) if !value.trim().is_empty() => Some(value),
_ => existing_secret.filter(|value| !value.trim().is_empty()),
};
let config = normalize_provider_config(OidcProviderConfig {
id: provider_id.to_string(),
enabled: input.enabled,
config_url: input.config_url,
issuer: input.issuer,
client_id: input.client_id,
client_secret,
scopes: input.scopes,
other_audiences: input.other_audiences,
redirect_uri: input.redirect_uri,
redirect_uri_dynamic: input.redirect_uri_dynamic,
claim_name: input.claim_name,
claim_prefix: input.claim_prefix,
role_policy: input.role_policy,
display_name: input.display_name,
groups_claim: input.groups_claim,
roles_claim: input.roles_claim,
email_claim: input.email_claim,
username_claim: input.username_claim,
hide_from_ui: input.hide_from_ui,
});
validate_provider_config_fields(&config)?;
Ok(config)
}
pub fn build_upsert_provider_config(
provider_id: &str,
input: OidcProviderConfigInput,
existing_secret: Option<String>,
) -> Result<OidcProviderConfig, OidcConfigError> {
build_provider_config(provider_id, input, existing_secret)
}
pub fn build_validation_provider_config(input: OidcProviderValidationInput) -> Result<OidcProviderConfig, OidcConfigError> {
let (provider_id, input) = input.into();
build_provider_config(&provider_id, input, None)
}
pub fn persisted_provider_secret(config: &ServerConfig, provider_id: &str) -> Option<String> {
config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|subsystem| subsystem.get(&provider_instance_key(provider_id)))
.and_then(|kvs| kvs.lookup(OIDC_CLIENT_SECRET))
.filter(|value| !value.trim().is_empty())
}
fn set_kvs_value(kvs: &mut KVS, key: &str, value: String) {
if let Some(existing) = kvs.0.iter_mut().find(|kv| kv.key == key) {
existing.value = value;
return;
}
kvs.insert(key.to_string(), value);
}
pub fn upsert_persisted_provider_config(config: &mut ServerConfig, provider: &OidcProviderConfig) {
let mut kvs = ServerConfig::new()
.get_value(IDENTITY_OPENID_SUB_SYS, DEFAULT_DELIMITER)
.unwrap_or_default();
set_kvs_value(
&mut kvs,
ENABLE_KEY,
if provider.enabled {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
set_kvs_value(&mut kvs, OIDC_CONFIG_URL, provider.config_url.clone());
set_kvs_value(&mut kvs, OIDC_ISSUER, provider.issuer.clone().unwrap_or_default());
set_kvs_value(&mut kvs, OIDC_CLIENT_ID, provider.client_id.clone());
set_kvs_value(&mut kvs, OIDC_CLIENT_SECRET, provider.client_secret.clone().unwrap_or_default());
set_kvs_value(&mut kvs, OIDC_SCOPES, provider.scopes.join(","));
set_kvs_value(&mut kvs, OIDC_OTHER_AUDIENCES, provider.other_audiences.join(","));
set_kvs_value(&mut kvs, OIDC_REDIRECT_URI, provider.redirect_uri.clone().unwrap_or_default());
set_kvs_value(
&mut kvs,
OIDC_REDIRECT_URI_DYNAMIC,
if provider.redirect_uri_dynamic {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
set_kvs_value(&mut kvs, OIDC_CLAIM_NAME, provider.claim_name.clone());
set_kvs_value(&mut kvs, OIDC_CLAIM_PREFIX, provider.claim_prefix.clone());
set_kvs_value(&mut kvs, OIDC_ROLE_POLICY, provider.role_policy.clone());
set_kvs_value(&mut kvs, OIDC_DISPLAY_NAME, provider.display_name.clone());
set_kvs_value(&mut kvs, OIDC_GROUPS_CLAIM, provider.groups_claim.clone());
set_kvs_value(&mut kvs, OIDC_ROLES_CLAIM, provider.roles_claim.clone());
set_kvs_value(&mut kvs, OIDC_EMAIL_CLAIM, provider.email_claim.clone());
set_kvs_value(&mut kvs, OIDC_USERNAME_CLAIM, provider.username_claim.clone());
set_kvs_value(
&mut kvs,
OIDC_HIDE_FROM_UI,
if provider.hide_from_ui {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
config
.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(provider_instance_key(&provider.id), kvs);
}
pub fn delete_persisted_provider_config(config: &mut ServerConfig, provider_id: &str) -> Result<(), OidcConfigError> {
let Some(subsystem) = config.0.get_mut(IDENTITY_OPENID_SUB_SYS) else {
return Err(OidcConfigError::ProviderNotFound);
};
if subsystem.remove(&provider_instance_key(provider_id)).is_none() {
return Err(OidcConfigError::ProviderNotFound);
}
if subsystem.is_empty() {
config.0.remove(IDENTITY_OPENID_SUB_SYS);
}
Ok(())
}
/// Parse all OIDC provider configs from environment variables.
pub(super) fn parse_env_configs() -> Vec<OidcProviderConfig> {
let mut configs = Vec::new();
// Check for the default provider (no suffix)
if let Some(config) = parse_single_provider("", "default") {
configs.push(config);
}
// Scan for suffixed providers by checking all OIDC env var prefixes.
// This allows providers to be discovered without requiring a separate ENABLE_ key.
let mut provider_ids: Vec<String> = Vec::new();
let scan_prefixes: Vec<String> = ENV_IDENTITY_OPENID_KEYS.iter().map(|k| format!("{k}_")).collect();
for (key, _) in std::env::vars() {
for prefix in &scan_prefixes {
if let Some(suffix) = key.strip_prefix(prefix.as_str())
&& !suffix.is_empty()
&& suffix != "default"
{
provider_ids.push(suffix.to_string());
break;
}
}
}
provider_ids.sort();
provider_ids.dedup();
for id in provider_ids {
let suffix = format!("_{id}");
if let Some(config) = parse_single_provider(&suffix, &id) {
configs.push(config);
}
}
configs
}
pub(super) fn parse_persisted_configs(cfg: &ServerConfig) -> Vec<OidcProviderConfig> {
let Some(subsystem) = cfg.0.get(IDENTITY_OPENID_SUB_SYS) else {
return Vec::new();
};
let mut configs = Vec::new();
let mut provider_ids: Vec<String> = subsystem.keys().cloned().collect();
provider_ids.sort();
for raw_id in provider_ids {
let Some(kvs) = subsystem.get(&raw_id) else {
continue;
};
let id = if raw_id == DEFAULT_DELIMITER {
"default"
} else {
raw_id.as_str()
};
if let Some(config) = parse_single_persisted_provider(kvs, id) {
configs.push(config);
}
}
configs
}
/// Parse a string as an `EnableState` boolean.
/// Returns `default_if_empty` when the input is empty, and `default_on_error`
/// when parsing fails.
pub(super) fn parse_enable_state(value: &str, default_if_empty: bool, default_on_error: bool) -> bool {
if value.is_empty() {
return default_if_empty;
}
value
.parse::<EnableState>()
.map(|s| s.is_enabled())
.unwrap_or(default_on_error)
}
/// Parse a single provider's config from env vars with the given suffix.
pub(super) fn parse_single_provider(env_suffix: &str, id: &str) -> Option<OidcProviderConfig> {
let get_env = |base: &str| -> String { std::env::var(format!("{base}{env_suffix}")).unwrap_or_default() };
let enable_val = get_env(ENV_IDENTITY_OPENID_ENABLE);
let config_url = get_env(ENV_IDENTITY_OPENID_CONFIG_URL);
let issuer = get_env(ENV_IDENTITY_OPENID_ISSUER);
// Skip if no config URL
if config_url.is_empty() {
return None;
}
let enabled = parse_enable_state(&enable_val, true, false);
let scopes_str = get_env(ENV_IDENTITY_OPENID_SCOPES);
let scopes = if scopes_str.is_empty() {
OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect()
} else {
scopes_str.split(',').map(|s| s.trim().to_string()).collect()
};
let other_audiences_str = get_env(ENV_IDENTITY_OPENID_OTHER_AUDIENCES);
let other_audiences = other_audiences_str
.split(',')
.map(|s| s.trim())
.filter(|s| !s.is_empty())
.map(|s| s.to_string())
.collect();
let redirect_uri_dynamic = parse_enable_state(&get_env(ENV_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC), true, true);
let claim_name = {
let v = get_env(ENV_IDENTITY_OPENID_CLAIM_NAME);
if v.is_empty() {
OIDC_DEFAULT_CLAIM_NAME.to_string()
} else {
v
}
};
let groups_claim = {
let v = get_env(ENV_IDENTITY_OPENID_GROUPS_CLAIM);
if v.is_empty() {
OIDC_DEFAULT_GROUPS_CLAIM.to_string()
} else {
v
}
};
let roles_claim = get_env(ENV_IDENTITY_OPENID_ROLES_CLAIM);
let email_claim = {
let v = get_env(ENV_IDENTITY_OPENID_EMAIL_CLAIM);
if v.is_empty() {
OIDC_DEFAULT_EMAIL_CLAIM.to_string()
} else {
v
}
};
let username_claim = {
let v = get_env(ENV_IDENTITY_OPENID_USERNAME_CLAIM);
if v.is_empty() {
OIDC_DEFAULT_USERNAME_CLAIM.to_string()
} else {
v
}
};
let display_name = {
let v = get_env(ENV_IDENTITY_OPENID_DISPLAY_NAME);
if v.is_empty() { id.to_string() } else { v }
};
let redirect_uri = {
let v = get_env(ENV_IDENTITY_OPENID_REDIRECT_URI);
if v.is_empty() { None } else { Some(v) }
};
let client_secret = {
let v = get_env(ENV_IDENTITY_OPENID_CLIENT_SECRET);
if v.is_empty() { None } else { Some(v) }
};
let hide_from_ui = parse_enable_state(&get_env(ENV_IDENTITY_OPENID_HIDE_FROM_UI), false, false);
Some(OidcProviderConfig {
id: id.to_string(),
enabled,
config_url,
issuer: if issuer.is_empty() { None } else { Some(issuer) },
client_id: get_env(ENV_IDENTITY_OPENID_CLIENT_ID),
client_secret,
scopes,
other_audiences,
redirect_uri,
redirect_uri_dynamic,
claim_name,
claim_prefix: get_env(ENV_IDENTITY_OPENID_CLAIM_PREFIX),
role_policy: get_env(ENV_IDENTITY_OPENID_ROLE_POLICY),
display_name,
groups_claim,
roles_claim,
email_claim,
username_claim,
hide_from_ui,
})
}
pub(super) fn parse_single_persisted_provider(kvs: &KVS, id: &str) -> Option<OidcProviderConfig> {
let config_url = kvs.get(OIDC_CONFIG_URL);
if config_url.is_empty() {
return None;
}
let enabled = parse_enable_state(&kvs.lookup(ENABLE_KEY).unwrap_or_default(), false, false);
let scopes_str = kvs.get(OIDC_SCOPES);
let scopes = if scopes_str.is_empty() {
OIDC_DEFAULT_SCOPES.split(',').map(String::from).collect()
} else {
scopes_str.split(',').map(|s| s.trim().to_string()).collect()
};
let other_audiences_str = kvs.get(OIDC_OTHER_AUDIENCES);
let other_audiences = other_audiences_str
.split(',')
.map(|s| s.trim())
.filter(|s| !s.is_empty())
.map(|s| s.to_string())
.collect();
let redirect_uri_dynamic = parse_enable_state(&kvs.lookup(OIDC_REDIRECT_URI_DYNAMIC).unwrap_or_default(), true, true);
let claim_name = kvs
.lookup(OIDC_CLAIM_NAME)
.unwrap_or_else(|| OIDC_DEFAULT_CLAIM_NAME.to_string());
let groups_claim = kvs
.lookup(OIDC_GROUPS_CLAIM)
.unwrap_or_else(|| OIDC_DEFAULT_GROUPS_CLAIM.to_string());
let roles_claim = kvs
.lookup(OIDC_ROLES_CLAIM)
.unwrap_or_else(|| OIDC_DEFAULT_ROLES_CLAIM.to_string());
let email_claim = kvs
.lookup(OIDC_EMAIL_CLAIM)
.unwrap_or_else(|| OIDC_DEFAULT_EMAIL_CLAIM.to_string());
let username_claim = kvs
.lookup(OIDC_USERNAME_CLAIM)
.unwrap_or_else(|| OIDC_DEFAULT_USERNAME_CLAIM.to_string());
let display_name = kvs.lookup(OIDC_DISPLAY_NAME).unwrap_or_else(|| id.to_string());
let redirect_uri = kvs.lookup(OIDC_REDIRECT_URI).filter(|v| !v.is_empty());
let client_secret = kvs.lookup(OIDC_CLIENT_SECRET).filter(|v| !v.is_empty());
let hide_from_ui = parse_enable_state(&kvs.lookup(OIDC_HIDE_FROM_UI).unwrap_or_default(), false, false);
Some(OidcProviderConfig {
id: id.to_string(),
enabled,
config_url,
issuer: kvs.lookup(OIDC_ISSUER).filter(|v| !v.is_empty()),
client_id: kvs.get(OIDC_CLIENT_ID),
client_secret,
scopes,
other_audiences,
redirect_uri,
redirect_uri_dynamic,
claim_name,
claim_prefix: kvs.get(OIDC_CLAIM_PREFIX),
role_policy: kvs.get(OIDC_ROLE_POLICY),
display_name,
groups_claim,
roles_claim,
email_claim,
username_claim,
hide_from_ui,
})
}
pub fn load_oidc_provider_configs_from_env() -> Vec<OidcProviderConfig> {
parse_env_configs()
}
pub fn load_oidc_provider_configs_from_server_config(cfg: &ServerConfig) -> Vec<OidcProviderConfig> {
parse_persisted_configs(cfg)
}
pub(super) fn merge_oidc_provider_configs(
env_configs: Vec<OidcProviderConfig>,
persisted_configs: Vec<OidcProviderConfig>,
) -> Vec<SourcedOidcProviderConfig> {
let mut effective = HashMap::new();
for config in persisted_configs {
effective.insert(
config.id.clone(),
SourcedOidcProviderConfig {
config,
source: OidcProviderConfigSource::Persisted,
},
);
}
for config in env_configs {
effective.insert(
config.id.clone(),
SourcedOidcProviderConfig {
config,
source: OidcProviderConfigSource::Env,
},
);
}
let mut configs: Vec<SourcedOidcProviderConfig> = effective.into_values().collect();
configs.sort_by(|lhs, rhs| lhs.config.id.cmp(&rhs.config.id));
configs
}
pub(super) fn load_effective_oidc_provider_configs(server_config: Option<&ServerConfig>) -> Vec<SourcedOidcProviderConfig> {
let env_configs = load_oidc_provider_configs_from_env();
let persisted_configs = server_config
.map(load_oidc_provider_configs_from_server_config)
.unwrap_or_default();
merge_oidc_provider_configs(env_configs, persisted_configs)
}
pub fn load_oidc_config_snapshot(server_config: Option<&ServerConfig>) -> OidcConfigSnapshot {
OidcConfigSnapshot::new(load_effective_oidc_provider_configs(server_config))
}
#[cfg(test)]
#[path = "config_tests.rs"]
mod tests;
crates/iam/src/oidc/config_tests.rs
+503
View File
@@ -0,0 +1,503 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::oidc::test_support::*;
use rustfs_config::server_config::{Config as ServerConfig, KVS};
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY};
#[test]
fn test_parse_single_provider_no_config_url() {
let config = parse_single_provider("_TEST_EMPTY", "test_empty");
assert!(config.is_none());
}
#[test]
fn test_parse_single_provider_reads_issuer() {
temp_env::with_vars(
[
(
ENV_IDENTITY_OPENID_CONFIG_URL,
Some("http://keycloak.ns.svc.cluster.local:8080/realms/app/.well-known/openid-configuration"),
),
(ENV_IDENTITY_OPENID_ISSUER, Some("https://app.local/realms/app")),
(ENV_IDENTITY_OPENID_CLIENT_ID, Some("console")),
],
|| {
let config = parse_single_provider("", "default").expect("provider config should parse");
assert_eq!(config.issuer.as_deref(), Some("https://app.local/realms/app"));
},
);
}
#[test]
fn test_parse_persisted_provider_config() {
let mut cfg = ServerConfig::new();
let mut kvs = KVS(vec![
rustfs_config::server_config::KV {
key: ENABLE_KEY.to_string(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CONFIG_URL.to_string(),
value: String::new(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CLIENT_ID.to_string(),
value: String::new(),
hidden_if_empty: false,
},
]);
kvs.insert(
OIDC_CONFIG_URL.to_string(),
"https://example.com/.well-known/openid-configuration".to_string(),
);
kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string());
kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string());
kvs.insert(OIDC_ISSUER.to_string(), "https://issuer.example".to_string());
kvs.insert(OIDC_ROLES_CLAIM.to_string(), "app_roles".to_string());
cfg.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(DEFAULT_DELIMITER.to_string(), kvs);
let parsed = parse_persisted_configs(&cfg);
assert_eq!(parsed.len(), 1);
assert_eq!(parsed[0].id, "default");
assert_eq!(parsed[0].client_id, "console");
assert_eq!(parsed[0].issuer.as_deref(), Some("https://issuer.example"));
assert!(parsed[0].enabled);
assert_eq!(parsed[0].roles_claim, "app_roles");
}
#[test]
fn test_parse_persisted_provider_config_omitted_roles_claim_is_empty() {
let mut cfg = ServerConfig::new();
let mut kvs = KVS(vec![
rustfs_config::server_config::KV {
key: ENABLE_KEY.to_string(),
value: EnableState::Off.to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CONFIG_URL.to_string(),
value: String::new(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CLIENT_ID.to_string(),
value: String::new(),
hidden_if_empty: false,
},
]);
kvs.insert(
OIDC_CONFIG_URL.to_string(),
"https://example.com/.well-known/openid-configuration".to_string(),
);
kvs.insert(OIDC_CLIENT_ID.to_string(), "console".to_string());
kvs.insert(ENABLE_KEY.to_string(), EnableState::On.to_string());
cfg.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(DEFAULT_DELIMITER.to_string(), kvs);
let parsed = parse_persisted_configs(&cfg);
assert_eq!(parsed.len(), 1);
assert_eq!(parsed[0].roles_claim, "");
}
#[test]
fn test_merge_oidc_provider_configs_prefers_env() {
let mut persisted = test_config("default");
persisted.display_name = "Persisted".to_string();
let mut env = test_config("default");
env.display_name = "Environment".to_string();
let merged = merge_oidc_provider_configs(vec![env], vec![persisted]);
assert_eq!(merged.len(), 1);
assert_eq!(merged[0].config.display_name, "Environment");
assert_eq!(merged[0].source, OidcProviderConfigSource::Env);
}
#[test]
fn test_oidc_provider_config_debug_redacts_client_secret() {
let config = OidcProviderConfig {
client_secret: Some("oidc-client-secret".to_string()),
..test_config("default")
};
let sourced = SourcedOidcProviderConfig {
config,
source: OidcProviderConfigSource::Persisted,
};
let rendered = format!("{sourced:?}");
assert!(!rendered.contains("oidc-client-secret"));
assert!(rendered.contains(REDACTED_SECRET));
assert!(rendered.contains("client-id"));
}
#[test]
fn test_parse_enable_state_on() {
assert!(parse_enable_state("on", false, false));
}
#[test]
fn test_parse_enable_state_off() {
assert!(!parse_enable_state("off", true, true));
}
#[test]
fn test_parse_enable_state_empty_returns_default() {
assert!(parse_enable_state("", true, false));
assert!(!parse_enable_state("", false, true));
}
#[test]
fn test_parse_enable_state_invalid_returns_error_default() {
assert!(!parse_enable_state("garbage", true, false));
assert!(parse_enable_state("garbage", false, true));
}
#[test]
fn test_hide_from_ui_default_is_false() {
let config = test_config("default");
assert!(!config.hide_from_ui);
}
#[test]
fn test_parse_persisted_hide_from_ui_off_is_false() {
let mut cfg = ServerConfig::new();
let mut kvs = KVS(vec![
rustfs_config::server_config::KV {
key: ENABLE_KEY.to_string(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CONFIG_URL.to_string(),
value: "https://example.com/.well-known/openid-configuration".to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CLIENT_ID.to_string(),
value: "console".to_string(),
hidden_if_empty: false,
},
]);
kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::Off.to_string());
cfg.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(DEFAULT_DELIMITER.to_string(), kvs);
let parsed = parse_persisted_configs(&cfg);
assert_eq!(parsed.len(), 1);
assert!(!parsed[0].hide_from_ui);
}
#[test]
fn test_parse_persisted_hide_from_ui_missing_defaults_false() {
let mut cfg = ServerConfig::new();
let kvs = KVS(vec![
rustfs_config::server_config::KV {
key: ENABLE_KEY.to_string(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CONFIG_URL.to_string(),
value: "https://example.com/.well-known/openid-configuration".to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CLIENT_ID.to_string(),
value: "console".to_string(),
hidden_if_empty: false,
},
]);
cfg.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(DEFAULT_DELIMITER.to_string(), kvs);
let parsed = parse_persisted_configs(&cfg);
assert_eq!(parsed.len(), 1);
assert!(!parsed[0].hide_from_ui);
}
#[test]
fn test_parse_persisted_hide_from_ui() {
let mut cfg = ServerConfig::new();
let mut kvs = KVS(vec![
rustfs_config::server_config::KV {
key: ENABLE_KEY.to_string(),
value: EnableState::On.to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CONFIG_URL.to_string(),
value: "https://example.com/.well-known/openid-configuration".to_string(),
hidden_if_empty: false,
},
rustfs_config::server_config::KV {
key: OIDC_CLIENT_ID.to_string(),
value: "console".to_string(),
hidden_if_empty: false,
},
]);
kvs.insert(OIDC_HIDE_FROM_UI.to_string(), EnableState::On.to_string());
cfg.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(DEFAULT_DELIMITER.to_string(), kvs);
let parsed = parse_persisted_configs(&cfg);
assert_eq!(parsed.len(), 1);
assert!(parsed[0].hide_from_ui);
}
#[test]
fn test_active_config_snapshot_puts_default_first() {
let sys = make_test_sys(vec![test_config("zeta"), test_config("default"), test_config("alpha")]);
assert_eq!(
sys.config_snapshot()
.providers()
.iter()
.map(|provider| provider.config.id.as_str())
.collect::<Vec<_>>(),
["default", "alpha", "zeta"]
);
}
#[test]
fn test_oidc_provider_config_defaults() {
let config = OidcProviderConfig {
id: "test".to_string(),
enabled: true,
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
issuer: None,
client_id: "my-client".to_string(),
client_secret: Some("secret".to_string()),
scopes: vec!["openid".to_string(), "profile".to_string(), "email".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: "groups".to_string(),
claim_prefix: "".to_string(),
role_policy: "readwrite".to_string(),
display_name: "Test Provider".to_string(),
groups_claim: "groups".to_string(),
roles_claim: String::new(),
email_claim: "email".to_string(),
username_claim: "preferred_username".to_string(),
hide_from_ui: false,
};
assert_eq!(config.id, "test");
assert!(config.enabled);
assert_eq!(config.scopes.len(), 3);
assert!(config.redirect_uri_dynamic);
}
#[test]
fn config_input_preserves_secret_and_defaults() {
let input = OidcProviderConfigInput {
config_url: " https://example.com/.well-known/openid-configuration ".to_string(),
client_id: " client ".to_string(),
client_secret: Some(String::new()),
scopes: vec![" openid ".to_string(), " profile ".to_string()],
roles_claim: "app_roles".to_string(),
..Default::default()
};
let config = build_upsert_provider_config("default", input, Some("existing-secret".to_string()))
.expect("valid config should preserve existing secret");
assert_eq!(config.client_secret.as_deref(), Some("existing-secret"));
assert_eq!(config.config_url, "https://example.com/.well-known/openid-configuration");
assert_eq!(config.client_id, "client");
assert_eq!(config.scopes, ["openid", "profile"]);
assert_eq!(config.roles_claim, "app_roles");
assert_eq!(config.claim_name, OIDC_DEFAULT_CLAIM_NAME);
let raw_secret = build_upsert_provider_config(
"default",
OidcProviderConfigInput {
config_url: config.config_url,
client_id: config.client_id,
client_secret: Some(" raw secret ".to_string()),
..Default::default()
},
None,
)
.expect("a nonempty client secret should be accepted");
assert_eq!(raw_secret.client_secret.as_deref(), Some(" raw secret "));
let input = OidcProviderConfigInput {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client-id".to_string(),
client_secret: Some("".to_string()),
..Default::default()
};
let config =
build_upsert_provider_config("default", input, Some("existing-secret".to_string())).expect("config should build");
assert_eq!(config.client_secret.as_deref(), Some("existing-secret"));
assert_eq!(config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM);
}
#[test]
fn config_input_rejects_invalid_fields() {
let input = OidcProviderConfigInput {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client".to_string(),
scopes: vec!["profile".to_string()],
..Default::default()
};
assert!(matches!(
build_upsert_provider_config("default", input, None),
Err(OidcConfigError::OpenidScopeRequired)
));
let input = OidcProviderConfigInput {
config_url: "https://127.0.0.1/.well-known/openid-configuration".to_string(),
client_id: "client".to_string(),
..Default::default()
};
assert!(matches!(
build_upsert_provider_config("default", input, None),
Err(OidcConfigError::ForbiddenOutbound { field: "config_url", .. })
));
assert!(serde_json::from_str::<OidcProviderConfigInput>(r#"{"unexpected_field":true}"#).is_err());
assert!(serde_json::from_str::<OidcProviderValidationInput>(r#"{"unexpected_field":true}"#).is_err());
let err = serde_json::from_str::<OidcProviderConfigInput>(
r#"{"config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#,
)
.expect_err("unknown upsert field should fail");
assert!(err.to_string().contains("unknown field"));
let err = serde_json::from_str::<OidcProviderValidationInput>(
r#"{"provider_id":"default","config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#,
)
.expect_err("unknown validate field should fail");
assert!(err.to_string().contains("unknown field"));
assert!(matches!(
validate_mutable_provider_id("../escape"),
Err(OidcConfigError::InvalidProviderId)
));
}
#[test]
fn validation_input_defaults_to_default_provider() {
let input = OidcProviderValidationInput {
provider_id: " ".to_string(),
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client".to_string(),
..Default::default()
};
let config = build_validation_provider_config(input).expect("validation input should use default provider");
assert_eq!(config.id, "default");
}
#[test]
fn persisted_config_codec_keeps_instance_keys_and_fields() {
assert_eq!(provider_instance_key("default"), "_");
assert_eq!(provider_instance_key("okta"), "okta");
let mut config = ServerConfig::new();
let mut provider = build_upsert_provider_config(
"default",
OidcProviderConfigInput {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
issuer: Some("https://issuer.example.com".to_string()),
client_id: "client".to_string(),
client_secret: Some("secret".to_string()),
hide_from_ui: true,
..Default::default()
},
None,
)
.expect("provider should be valid");
upsert_persisted_provider_config(&mut config, &provider);
let default = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|subsystem| subsystem.get(DEFAULT_DELIMITER))
.expect("default provider KVS should exist");
assert_eq!(default.get(OIDC_ISSUER), "https://issuer.example.com");
assert_eq!(default.get(OIDC_HIDE_FROM_UI), EnableState::On.to_string());
assert_eq!(persisted_provider_secret(&config, "default").as_deref(), Some("secret"));
provider.hide_from_ui = false;
upsert_persisted_provider_config(&mut config, &provider);
let default = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|subsystem| subsystem.get(DEFAULT_DELIMITER))
.expect("updated default provider KVS should exist");
assert_eq!(default.get(OIDC_HIDE_FROM_UI), EnableState::Off.to_string());
delete_persisted_provider_config(&mut config, "default").expect("default provider should be deleted");
assert!(matches!(
delete_persisted_provider_config(&mut config, "default"),
Err(OidcConfigError::ProviderNotFound)
));
let mut named_provider = OidcProviderConfig {
issuer: Some("https://app.local/realms/app".to_string()),
hide_from_ui: true,
..test_config("kubernetes")
};
upsert_persisted_provider_config(&mut config, &named_provider);
let kvs = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|providers| providers.get("kubernetes"))
.expect("provider KVS should exist");
assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::On.to_string());
assert_eq!(kvs.get(OIDC_ISSUER), "https://app.local/realms/app");
named_provider.hide_from_ui = false;
upsert_persisted_provider_config(&mut config, &named_provider);
let kvs = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|providers| providers.get("kubernetes"))
.expect("provider KVS should exist");
assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::Off.to_string());
}
#[test]
fn site_replication_snapshot_hashes_secret_without_exposing_config() {
let provider = build_upsert_provider_config(
"default",
OidcProviderConfigInput {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client".to_string(),
client_secret: Some("secret".to_string()),
..Default::default()
},
None,
)
.expect("provider should be valid");
let snapshot = OidcSiteReplicationSnapshot::from_config_snapshot(&OidcConfigSnapshot::new(vec![SourcedOidcProviderConfig {
config: provider,
source: OidcProviderConfigSource::Persisted,
}]));
assert_eq!(
snapshot.providers()[0].hashed_client_secret,
"K7gNU3sdo-OL0wNhqoVWhr3g6s1xYv72ol_pe_Unols"
);
assert!(!format!("{snapshot:?}").contains("\"secret\""));
assert_eq!(hash_client_secret(None), "");
assert_eq!(hash_client_secret(Some("")), "");
}
crates/iam/src/oidc/mod.rs
+39
View File
@@ -0,0 +1,39 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! OpenID Connect configuration, provider discovery, runtime flows, and state.
mod config;
mod provider;
mod runtime;
mod state;
mod transport;
#[cfg(any(test, feature = "test-util"))]
mod test_support;
pub use config::{
OidcConfigError, OidcConfigQuery, OidcConfigSnapshot, OidcProviderConfig, OidcProviderConfigInput, OidcProviderConfigSource,
OidcProviderValidationInput, OidcProviderValidationResult, OidcSiteReplicationProvider, OidcSiteReplicationSnapshot,
SourcedOidcProviderConfig, build_upsert_provider_config, build_validation_provider_config, delete_persisted_provider_config,
load_oidc_config_snapshot, load_oidc_provider_configs_from_env, load_oidc_provider_configs_from_server_config,
persisted_provider_secret, upsert_persisted_provider_config, validate_mutable_provider_id,
};
pub use provider::validate_oidc_provider_config_with_extra_root_ca;
pub use runtime::{OidcSys, StandardOidcAdapter};
pub use transport::{
OidcExtraRootCaMaterial, OidcExtraRootCaProvider, OidcPluginAuthnMetricsSnapshot, oidc_plugin_authn_metrics_snapshot,
};
#[cfg(any(test, feature = "test-util"))]
pub use test_support::{make_test_sys, test_config};
crates/iam/src/oidc/provider.rs
+632
View File
@@ -0,0 +1,632 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::{
config::{OidcProviderConfig, OidcProviderValidationResult, SourcedOidcProviderConfig},
transport::{OidcHttpError, ReqwestHttpClient, oidc_extra_root_certs},
};
use crate::{EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC};
use openidconnect::core::{CoreIdTokenVerifier, CoreJsonWebKeySet, CoreJwsSigningAlgorithm};
use openidconnect::{
AsyncHttpClient, ClientId, ClientSecret, DiscoveryError, IssuerUrl, JsonWebKeySetUrl, ProviderMetadataWithLogout, TokenUrl,
};
use serde::Deserialize;
use std::collections::HashMap;
use std::sync::RwLock;
use std::time::{Duration as StdDuration, Instant};
use tokio::time::sleep;
use tracing::warn;
use url::Url;
pub(super) const OIDC_JWKS_REFRESH_INTERVAL: StdDuration = StdDuration::from_secs(24 * 60 * 60);
pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRIES: usize = 3;
pub(super) const OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY: StdDuration = StdDuration::from_millis(50);
pub(super) const OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY: &str = "JWKS request blocked by outbound policy";
pub(super) const OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY: &str = "OIDC provider discovery blocked by outbound policy";
// ---- Internal provider state ----
/// Discovered OIDC provider metadata.
/// We store metadata (which includes JWKS after discovery) separately rather than
/// a `CoreClient` because the crate uses type-state generics that make storing
/// the configured client in a HashMap impractical. The client is reconstructed
/// on-the-fly from metadata when needed.
#[derive(Clone)]
pub(super) struct ProviderState {
pub(super) metadata: DiscoveredProviderMetadata,
pub(super) discovered_at: Instant,
}
pub(super) struct ProviderRuntime {
states: RwLock<HashMap<String, ProviderState>>,
http_client: ReqwestHttpClient,
}
// Workload issuers do not implement the browser authorization flow. Keep their
// verification metadata separate rather than inventing an authorization URL.
#[serde_with::serde_as]
#[derive(Clone, Deserialize)]
pub(super) struct WorkloadProviderMetadata {
issuer: IssuerUrl,
jwks_uri: JsonWebKeySetUrl,
token_endpoint: Option<TokenUrl>,
#[serde_as(as = "serde_with::VecSkipError<_>")]
id_token_signing_alg_values_supported: Vec<CoreJwsSigningAlgorithm>,
#[serde(skip)]
jwks: CoreJsonWebKeySet,
// Discovery is extensible; report unsupported fields without logging values.
#[serde(flatten)]
additional_fields: HashMap<String, serde_json::Value>,
}
#[derive(Clone)]
pub(super) enum DiscoveredProviderMetadata {
Console(Box<ProviderMetadataWithLogout>),
Workload(Box<WorkloadProviderMetadata>),
}
impl DiscoveredProviderMetadata {
pub(super) fn parse(body: &[u8], hide_from_ui: bool) -> Result<Self, String> {
let document: serde_json::Value = serde_json::from_slice(body).map_err(|err| err.to_string())?;
if hide_from_ui
&& document
.as_object()
.is_some_and(|fields| !fields.contains_key("authorization_endpoint"))
{
let mut metadata: WorkloadProviderMetadata = serde_json::from_slice(body).map_err(|err| err.to_string())?;
if !metadata.additional_fields.is_empty() {
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "workload_discovery_additional_fields",
field_count = metadata.additional_fields.len(),
"workload discovery contains additional fields"
);
metadata.additional_fields.clear();
}
Ok(Self::Workload(Box::new(metadata)))
} else {
serde_json::from_slice(body)
.map(|metadata| Self::Console(Box::new(metadata)))
.map_err(|err| err.to_string())
}
}
pub(super) fn console(&self) -> Result<&ProviderMetadataWithLogout, String> {
match self {
Self::Console(metadata) => Ok(metadata),
Self::Workload(_) => Err("OIDC provider has no authorization endpoint; only web identity is supported".into()),
}
}
pub(super) fn issuer(&self) -> &IssuerUrl {
match self {
Self::Console(metadata) => metadata.issuer(),
Self::Workload(metadata) => &metadata.issuer,
}
}
pub(super) fn jwks_uri(&self) -> &JsonWebKeySetUrl {
match self {
Self::Console(metadata) => metadata.jwks_uri(),
Self::Workload(metadata) => &metadata.jwks_uri,
}
}
pub(super) fn set_jwks(self, jwks: CoreJsonWebKeySet) -> Self {
match self {
Self::Console(metadata) => Self::Console(Box::new(metadata.set_jwks(jwks))),
Self::Workload(mut metadata) => {
metadata.jwks = jwks;
Self::Workload(metadata)
}
}
}
pub(super) fn authorization_endpoint(&self) -> Option<String> {
match self {
Self::Console(metadata) => Some(metadata.authorization_endpoint().to_string()),
Self::Workload(_) => None,
}
}
pub(super) fn token_endpoint(&self) -> Option<&TokenUrl> {
match self {
Self::Console(metadata) => metadata.token_endpoint(),
Self::Workload(metadata) => metadata.token_endpoint.as_ref(),
}
}
pub(super) fn verifier(&self, config: &OidcProviderConfig) -> CoreIdTokenVerifier<'static> {
let client_id = ClientId::new(config.client_id.clone());
let secret = config.client_secret.as_ref().map(|secret| ClientSecret::new(secret.clone()));
let (issuer, jwks, algorithms) = match self {
Self::Console(metadata) => (metadata.issuer(), metadata.jwks(), metadata.id_token_signing_alg_values_supported()),
Self::Workload(metadata) => (&metadata.issuer, &metadata.jwks, &metadata.id_token_signing_alg_values_supported),
};
let verifier = match secret {
Some(secret) => CoreIdTokenVerifier::new_confidential_client(client_id, secret, issuer.clone(), jwks.clone()),
None => CoreIdTokenVerifier::new_public_client(client_id, issuer.clone(), jwks.clone()),
};
verifier.set_allowed_algs(algorithms.clone())
}
}
// This adapter is used only for discovery/JWKS fetches, never token exchange.
struct JwksAcceptClient<'a> {
inner: &'a ReqwestHttpClient,
discovery_url: Option<Url>,
}
impl<'c> AsyncHttpClient<'c> for JwksAcceptClient<'_> {
type Error = OidcHttpError;
type Future = <ReqwestHttpClient as AsyncHttpClient<'c>>::Future;
fn call(&'c self, mut request: http::Request<Vec<u8>>) -> Self::Future {
if !self.discovery_url.as_ref().is_some_and(|url| request.uri() == url.as_str()) {
request.headers_mut().insert(
http::header::ACCEPT,
http::HeaderValue::from_static("application/json, application/jwk-set+json"),
);
}
self.inner.call(request)
}
}
impl ProviderState {
fn is_stale(&self) -> bool {
self.discovered_at.elapsed() >= OIDC_JWKS_REFRESH_INTERVAL
}
}
impl ProviderRuntime {
pub(super) fn new(http_client: ReqwestHttpClient, states: HashMap<String, ProviderState>) -> Self {
Self {
states: RwLock::new(states),
http_client,
}
}
pub(super) fn http_client(&self) -> &ReqwestHttpClient {
&self.http_client
}
/// Find a provider whose discovered issuer matches the given JWT issuer string.
pub(super) fn find_provider_by_issuer(
&self,
issuer: &str,
configs: &HashMap<String, SourcedOidcProviderConfig>,
) -> Option<(String, OidcProviderConfig, ProviderState)> {
let (issuer_scheme, issuer_host, issuer_port, issuer_path) = normalize_issuer(issuer)?;
let map = self
.states
.read()
.map_err(|e| format!("provider state lock poisoned: {e}"))
.ok()?;
for (id, state) in map.iter() {
let provider_issuer = state.metadata.issuer().as_str();
let Some((provider_scheme, provider_host, provider_port, provider_path)) = normalize_issuer(provider_issuer) else {
continue;
};
if issuer_scheme == provider_scheme
&& issuer_host == provider_host
&& issuer_port == provider_port
&& issuer_path == provider_path
&& let Some(config) = configs.get(id)
{
return Some((id.clone(), config.config.clone(), state.clone()));
}
}
None
}
pub(super) fn get_provider_state(&self, provider_id: &str) -> Result<ProviderState, String> {
self.states
.read()
.map_err(|e| format!("provider state lock poisoned: {e}"))?
.get(provider_id)
.cloned()
.ok_or_else(|| format!("provider not discovered: {provider_id}"))
}
pub(super) async fn refresh_provider_state(
&self,
provider_id: &str,
config: &OidcProviderConfig,
) -> Result<ProviderState, String> {
let state = discover_provider(config, &self.http_client).await?;
let mut map = self.states.write().map_err(|e| {
let msg = e.to_string();
format!("provider state lock poisoned: {msg}")
})?;
map.insert(provider_id.to_string(), state.clone());
Ok(state)
}
pub(super) async fn ensure_provider_state(
&self,
provider_id: &str,
config: &OidcProviderConfig,
) -> Result<ProviderState, String> {
let state = self.get_provider_state(provider_id)?;
if state.is_stale() {
self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| {
warn!(
"OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}",
provider_id, refresh_err
);
Ok(state)
})
} else {
Ok(state)
}
}
pub(super) async fn ensure_provider_state_if_stale(
&self,
provider_id: &str,
config: &OidcProviderConfig,
state: &ProviderState,
) -> Result<ProviderState, String> {
if state.is_stale() {
self.refresh_provider_state(provider_id, config).await.or_else(|refresh_err| {
warn!(
"OIDC provider '{}' JWKS metadata refresh skipped due to transient network issue: {}",
provider_id, refresh_err
);
Ok(state.clone())
})
} else {
Ok(state.clone())
}
}
}
/// Perform OIDC discovery for a provider.
/// `discover_async` fetches the discovery document and JWKS in one step.
pub(super) async fn discover_provider(
config: &OidcProviderConfig,
http_client: &ReqwestHttpClient,
) -> Result<ProviderState, String> {
if let Some(issuer) = config.issuer.as_deref().filter(|issuer| !issuer.trim().is_empty()) {
return discover_provider_from_config_url(config, issuer, http_client).await;
}
// The openidconnect crate expects the issuer URL (base), not the
// .well-known/openid-configuration URL.
let base_issuer = normalize_config_url(&config.config_url)?;
let candidates = issuer_candidates(&base_issuer);
let mut last_errors = Vec::new();
for candidate_issuer in candidates.iter() {
let issuer_url = IssuerUrl::new(candidate_issuer.clone()).map_err(|e| format!("invalid issuer URL: {e}"))?;
for attempt in 0..OIDC_DISCOVERY_TRANSPORT_RETRIES {
let discovered = if config.hide_from_ui {
discover_provider_from_config_url(config, candidate_issuer, http_client).await
} else {
let client = JwksAcceptClient {
inner: http_client,
discovery_url: Some(
issuer_url
.join(".well-known/openid-configuration")
.map_err(|err| err.to_string())?,
),
};
ProviderMetadataWithLogout::discover_async(issuer_url.clone(), &client)
.await
.map(|metadata| ProviderState {
metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)),
discovered_at: Instant::now(),
})
.map_err(|err| match err {
DiscoveryError::Request(OidcHttpError::ForbiddenOutbound(reason)) => {
format!("{OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY}: {reason}")
}
err => format!("discovery failed: {err}"),
})
};
match discovered {
Ok(state) => return Ok(state),
Err(error)
if error.starts_with(OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY)
|| error.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY) =>
{
return Err(error);
}
Err(error) => {
let is_transient_transport = error.contains("Request failed");
let should_retry = is_transient_transport && attempt + 1 < OIDC_DISCOVERY_TRANSPORT_RETRIES;
if should_retry {
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "provider_discovery_transport_retry",
provider_id = %config.id,
config_url = %config.config_url,
issuer_candidate = %candidate_issuer,
attempt = attempt + 1,
max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES,
error = %error,
"oidc provider discovery failed"
);
sleep(OIDC_DISCOVERY_TRANSPORT_RETRY_DELAY).await;
continue;
}
last_errors.push(format!("issuer '{candidate_issuer}': {error}"));
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "provider_discovery_candidate_failed",
provider_id = %config.id,
config_url = %config.config_url,
issuer_candidate = %candidate_issuer,
attempt = attempt + 1,
max_attempts = OIDC_DISCOVERY_TRANSPORT_RETRIES,
error = %error,
"oidc provider discovery failed"
);
break;
}
}
}
}
Err(format!(
"discovery failed for all issuer variants {:?}: {}",
candidates,
last_errors.join("; ")
))
}
pub(super) async fn validate_oidc_provider_config(config: &OidcProviderConfig) -> Result<OidcProviderValidationResult, String> {
let http_client = ReqwestHttpClient::new()?;
validate_oidc_provider_config_with_http_client(config, &http_client).await
}
pub async fn validate_oidc_provider_config_with_extra_root_ca(
config: &OidcProviderConfig,
root_ca_pem: Option<&[u8]>,
) -> Result<OidcProviderValidationResult, String> {
if root_ca_pem.is_none() {
return validate_oidc_provider_config(config).await;
}
let http_client = ReqwestHttpClient::new_with_extra_root_certs(oidc_extra_root_certs(root_ca_pem)?)?;
validate_oidc_provider_config_with_http_client(config, &http_client).await
}
async fn validate_oidc_provider_config_with_http_client(
config: &OidcProviderConfig,
http_client: &ReqwestHttpClient,
) -> Result<OidcProviderValidationResult, String> {
let state = discover_provider(config, http_client).await?;
Ok(OidcProviderValidationResult {
issuer: state.metadata.issuer().to_string(),
authorization_endpoint: state.metadata.authorization_endpoint(),
token_endpoint: state.metadata.token_endpoint().map(ToString::to_string),
})
}
async fn discover_provider_from_config_url(
config: &OidcProviderConfig,
issuer: &str,
http_client: &ReqwestHttpClient,
) -> Result<ProviderState, String> {
let issuer_url = IssuerUrl::new(issuer.trim().to_string()).map_err(|e| format!("invalid issuer URL: {e}"))?;
let explicit_issuer = config.issuer.as_deref().is_some_and(|issuer| !issuer.trim().is_empty());
let discovery_url = if explicit_issuer {
discovery_url_from_config_url(&config.config_url)?
} else {
issuer_url
.join(".well-known/openid-configuration")
.map_err(|err| err.to_string())?
};
let request = http::Request::builder()
.uri(discovery_url.to_string())
.method(http::Method::GET)
.header(http::header::ACCEPT, "application/json")
.body(Vec::new())
.map_err(|err| format!("failed to prepare discovery request: {err}"))?;
let response = match http_client.call(request).await {
Ok(response) => response,
Err(OidcHttpError::ForbiddenOutbound(reason)) => {
return Err(format!("{OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY}: {reason}"));
}
Err(err) => return Err(format!("discovery request failed: Request failed: {err}")),
};
if response.status() != http::StatusCode::OK {
return Err(format!("discovery failed: HTTP status code {} at {}", response.status(), discovery_url));
}
if !explicit_issuer
&& let Some(content_type) = response.headers().get(http::header::CONTENT_TYPE)
&& !content_type.to_str().ok().is_some_and(|value| {
value
.split(';')
.next()
.is_some_and(|essence| essence.eq_ignore_ascii_case("application/json"))
})
{
return Err("Unexpected response Content-Type: expected application/json".into());
}
let provider_metadata = DiscoveredProviderMetadata::parse(response.body(), config.hide_from_ui)
.map_err(|err| format!("failed to parse discovery response: {err}"))?;
if provider_metadata.issuer() != &issuer_url {
return Err(format!(
"unexpected issuer URI `{}` (expected `{}`)",
provider_metadata.issuer().as_str(),
issuer_url.as_str()
));
}
let jwks_url = if explicit_issuer {
jwks_url_from_config_url(&config.config_url, &issuer_url, provider_metadata.jwks_uri())?
} else {
provider_metadata.jwks_uri().clone()
};
let jwks = match CoreJsonWebKeySet::fetch_async(
&jwks_url,
&JwksAcceptClient {
inner: http_client,
discovery_url: None,
},
)
.await
{
Ok(jwks) => jwks,
Err(DiscoveryError::Request(OidcHttpError::ForbiddenOutbound(reason))) => {
return Err(format!("{OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY}: {reason}"));
}
Err(err) => return Err(format!("failed to fetch JWKS: {err}")),
};
Ok(ProviderState {
metadata: provider_metadata.set_jwks(jwks),
discovered_at: Instant::now(),
})
}
pub(super) fn normalize_issuer(raw: &str) -> Option<(String, String, u16, String)> {
let parsed = Url::parse(raw).ok()?;
if parsed.scheme() != "http" && parsed.scheme() != "https" {
return None;
}
let host = parsed.host_str()?.to_ascii_lowercase();
let port = parsed.port_or_known_default()?;
let normalized_path = {
let path = parsed.path().trim_end_matches('/').to_string();
if path.is_empty() { "/".to_string() } else { path }
};
Some((parsed.scheme().to_string(), host, port, normalized_path))
}
pub(super) fn normalize_config_url(config_url: &str) -> Result<String, String> {
let config_url = config_url.trim();
let url = Url::parse(config_url).map_err(|e| format!("invalid config_url: {e}"))?;
if url.scheme() != "http" && url.scheme() != "https" {
return Err(format!("invalid config_url scheme: {}", url.scheme()));
}
let host = url.host_str().ok_or_else(|| "config_url missing host".to_string())?;
let path = url.path();
// Strip `/.well-known/openid-configuration` (with optional trailing slash) if present.
// Everything else is preserved exactly so the issuer URL matches the provider's discovery
// document (e.g. Authentik includes a trailing slash, Keycloak does not).
let normalized_path = path
.strip_suffix('/')
.unwrap_or(path)
.strip_suffix("/.well-known/openid-configuration")
.unwrap_or(if path == "/" { "" } else { path });
if normalized_path.contains("/.well-known/") {
return Err("config_url uses an unsupported .well-known discovery URL".into());
}
let mut issuer = format!("{}://{host}", url.scheme());
if let Some(port) = url.port() {
issuer.push(':');
issuer.push_str(&port.to_string());
}
if !normalized_path.is_empty() {
issuer.push_str(normalized_path);
}
Ok(issuer)
}
pub(super) fn discovery_url_from_config_url(config_url: &str) -> Result<Url, String> {
let mut url = Url::parse(config_url.trim()).map_err(|e| format!("invalid config_url: {e}"))?;
if url.scheme() != "http" && url.scheme() != "https" {
return Err(format!("invalid config_url scheme: {}", url.scheme()));
}
if url.host_str().is_none() {
return Err("config_url missing host".to_string());
}
let path = url.path().to_string();
let without_trailing_slash = path.strip_suffix('/').unwrap_or(&path);
if without_trailing_slash.ends_with("/.well-known/openid-configuration") {
url.set_path(without_trailing_slash);
return Ok(url);
}
if without_trailing_slash.contains("/.well-known/") {
return Err("config_url uses an unsupported .well-known discovery URL".into());
}
let discovery_path = if without_trailing_slash.is_empty() || without_trailing_slash == "/" {
"/.well-known/openid-configuration".to_string()
} else {
format!("{without_trailing_slash}/.well-known/openid-configuration")
};
url.set_path(&discovery_path);
Ok(url)
}
pub(super) fn jwks_url_from_config_url(
config_url: &str,
issuer_url: &IssuerUrl,
jwks_url: &JsonWebKeySetUrl,
) -> Result<JsonWebKeySetUrl, String> {
let issuer = issuer_url.url();
let jwks = jwks_url.url();
if issuer.origin() != jwks.origin() {
return Ok(jwks_url.clone());
}
let issuer_path = issuer.path().trim_end_matches('/');
let Some(suffix) = jwks.path().strip_prefix(issuer_path) else {
return Ok(jwks_url.clone());
};
if !suffix.is_empty() && !suffix.starts_with('/') {
return Ok(jwks_url.clone());
}
let mut internal_url =
Url::parse(&normalize_config_url(config_url)?).map_err(|err| format!("invalid config_url issuer base: {err}"))?;
let internal_path = internal_url.path().trim_end_matches('/');
internal_url.set_path(&format!("{internal_path}{suffix}"));
internal_url.set_query(jwks.query());
Ok(JsonWebKeySetUrl::from_url(internal_url))
}
pub(super) fn issuer_candidates(base: &str) -> Vec<String> {
let original = base.trim();
let mut variants = Vec::with_capacity(2);
variants.push(original.to_string());
let toggled = if original.ends_with('/') {
original.trim_end_matches('/').to_string()
} else {
format!("{original}/")
};
variants.push(toggled);
variants
}
#[cfg(test)]
#[path = "provider_tests.rs"]
mod tests;
crates/iam/src/oidc/provider_tests.rs
+712
View File
@@ -0,0 +1,712 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::oidc::{runtime::*, state::*, test_support::*, transport::*};
use jsonwebtoken::{Algorithm, EncodingKey, Header};
use openidconnect::core::CoreIdToken;
use openidconnect::{IssuerUrl, JsonWebKeySetUrl, Nonce};
use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy};
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::{Arc, Mutex};
use url::Url;
#[test]
fn test_normalize_issuer_matches() {
let lhs = normalize_issuer("https://idp.example.com/.well-known/openid-configuration/").unwrap();
let rhs = normalize_issuer("https://idp.example.com/.well-known/openid-configuration").unwrap();
assert_eq!(lhs, rhs);
assert_eq!(
lhs,
(
"https".to_string(),
"idp.example.com".to_string(),
443,
"/.well-known/openid-configuration".to_string()
)
);
}
#[test]
fn test_normalize_config_url() {
// --- Well-known suffix stripping ---
// Bare well-known URL → stripped to just the host
assert_eq!(
normalize_config_url("https://idp.example.com/.well-known/openid-configuration").unwrap(),
"https://idp.example.com"
);
// Trailing slash after well-known suffix is also stripped
assert_eq!(
normalize_config_url("https://idp.example.com/.well-known/openid-configuration/").unwrap(),
"https://idp.example.com"
);
// Well-known under a sub-path (Keycloak realms)
assert_eq!(
normalize_config_url("https://keycloak.example.com/realms/myrealm/.well-known/openid-configuration").unwrap(),
"https://keycloak.example.com/realms/myrealm"
);
// --- Providers WITHOUT trailing slash (Keycloak, Auth0, Okta, Google) ---
assert_eq!(
normalize_config_url("https://keycloak.example.com/realms/myrealm").unwrap(),
"https://keycloak.example.com/realms/myrealm"
);
assert_eq!(
normalize_config_url("https://idp.example.com/custom/realm").unwrap(),
"https://idp.example.com/custom/realm"
);
// --- Providers WITH trailing slash (Authentik) ---
assert_eq!(
normalize_config_url("https://auth.example.com/application/o/myapp/").unwrap(),
"https://auth.example.com/application/o/myapp/"
);
// --- Root-level issuer (bare host) ---
assert_eq!(normalize_config_url("https://idp.example.com").unwrap(), "https://idp.example.com");
assert_eq!(normalize_config_url("https://idp.example.com/").unwrap(), "https://idp.example.com");
// --- Custom port ---
assert_eq!(
normalize_config_url("https://idp.example.com:8443/auth/realms/test").unwrap(),
"https://idp.example.com:8443/auth/realms/test"
);
assert_eq!(
normalize_config_url("http://localhost:8080/application/o/app/").unwrap(),
"http://localhost:8080/application/o/app/"
);
// --- Error cases ---
assert!(normalize_config_url("https://idp.example.com/.well-known/invalid").is_err());
assert!(normalize_config_url("gopher://idp.example.com").is_err());
assert!(normalize_config_url("not-a-url").is_err());
}
#[test]
fn test_discovery_url_from_config_url() {
assert_eq!(
discovery_url_from_config_url("https://idp.example.com/.well-known/openid-configuration")
.expect("config URL should parse")
.as_str(),
"https://idp.example.com/.well-known/openid-configuration"
);
assert_eq!(
discovery_url_from_config_url("https://idp.example.com/realms/app")
.expect("issuer URL should derive discovery URL")
.as_str(),
"https://idp.example.com/realms/app/.well-known/openid-configuration"
);
assert!(discovery_url_from_config_url("https://idp.example.com/.well-known/not-openid").is_err());
}
#[test]
fn test_issuer_candidates() {
assert_eq!(
issuer_candidates("https://idp.example.com/realm"),
vec![
"https://idp.example.com/realm".to_string(),
"https://idp.example.com/realm/".to_string()
]
);
assert_eq!(
issuer_candidates("https://idp.example.com/realm/"),
vec![
"https://idp.example.com/realm/".to_string(),
"https://idp.example.com/realm".to_string()
]
);
assert_eq!(
issuer_candidates("https://idp.example.com"),
vec!["https://idp.example.com".to_string(), "https://idp.example.com/".to_string()]
);
}
#[test]
fn workload_metadata_requires_hidden_provider_and_valid_verification_fields() {
let document = serde_json::json!({
"issuer": "https://issuer.example.com",
"jwks_uri": "https://issuer.example.com/jwks",
"response_types_supported": ["id_token"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["ES256"],
});
let parse =
|value: &serde_json::Value, hidden| DiscoveredProviderMetadata::parse(&serde_json::to_vec(value).unwrap(), hidden);
let metadata = parse(&document, true).expect("hidden workload metadata should parse");
assert!(metadata.authorization_endpoint().is_none());
assert!(metadata.console().err().unwrap().contains("only web identity"));
assert!(parse(&document, false).err().unwrap().contains("authorization_endpoint"));
for field in ["issuer", "jwks_uri", "id_token_signing_alg_values_supported"] {
let mut invalid = document.clone();
invalid.as_object_mut().unwrap().remove(field);
assert!(parse(&invalid, true).err().unwrap().contains(field), "missing {field}");
}
for endpoint in [serde_json::Value::Null, serde_json::json!(""), serde_json::json!("not a URL")] {
let mut invalid = document.clone();
invalid["authorization_endpoint"] = endpoint;
assert!(parse(&invalid, true).is_err(), "invalid endpoint must not select workload metadata");
}
let mut complete = document;
complete["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize");
for hidden in [true, false] {
let metadata = parse(&complete, hidden).expect("full providers keep the existing parser");
assert!(metadata.console().is_ok());
assert!(metadata.token_endpoint().is_none(), "token endpoint remains optional");
}
}
#[test]
fn workload_metadata_rejects_duplicate_fields() {
for hidden in [false, true] {
let document = format!(
r#"{{"issuer":"https://wrong.example.com","issuer":"https://issuer.example.com",{}"jwks_uri":"https://issuer.example.com/jwks","response_types_supported":["id_token"],"subject_types_supported":["public"],"id_token_signing_alg_values_supported":["ES256"]}}"#,
if hidden {
""
} else {
r#""authorization_endpoint":"https://issuer.example.com/authorize","#
},
);
let error = DiscoveredProviderMetadata::parse(document.as_bytes(), hidden)
.err()
.expect("duplicate issuer must fail");
assert!(error.contains("duplicate field"), "{error}");
}
}
#[test]
fn workload_verifier_preserves_algorithm_secret_and_audience_policy() {
let secret = Nonce::new_random().secret().to_string();
let mut config = build_mocked_oidc_provider_config("workload", "https://issuer.example.com");
config.client_secret = Some(secret.clone());
config.other_audiences = vec!["additional-audience".into()];
let now = time::OffsetDateTime::now_utc().unix_timestamp();
let payload = serde_json::json!({
"iss": config.config_url, "sub": "repo:example/project:ref:refs/heads/main",
"aud": [config.client_id, "additional-audience"], "iat": now, "exp": now + 300,
});
let signed =
jsonwebtoken::encode(&Header::new(Algorithm::HS256), &payload, &EncodingKey::from_secret(secret.as_bytes())).unwrap();
let token: CoreIdToken = signed.parse().unwrap();
for workload in [false, true] {
for (algorithms, accepted) in [
(serde_json::json!(["HS256", "unsupported-future-algorithm"]), true),
(serde_json::json!(["ES256"]), false),
(serde_json::json!(["unsupported-future-algorithm"]), false),
] {
let mut document = serde_json::json!({
"issuer": config.config_url, "jwks_uri": "https://issuer.example.com/jwks",
"response_types_supported": ["id_token"], "subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": algorithms,
});
if !workload {
document["authorization_endpoint"] = serde_json::json!("https://issuer.example.com/authorize");
}
let metadata = DiscoveredProviderMetadata::parse(&serde_json::to_vec(&document).unwrap(), true).unwrap();
let verifier = metadata
.verifier(&config)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
assert_eq!(
token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_ok(),
accepted,
"workload={workload}, algorithms={algorithms}"
);
if accepted {
assert!(
token.claims(&metadata.verifier(&config), |_: Option<&Nonce>| Ok(())).is_err(),
"additional audiences require explicit trust"
);
let mut wrong_secret = config.clone();
wrong_secret.client_secret = Some(Nonce::new_random().secret().to_string());
let verifier = metadata
.verifier(&wrong_secret)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
assert!(
token.claims(&verifier, |_: Option<&Nonce>| Ok(())).is_err(),
"incorrect client secret must fail"
);
}
}
}
}
#[tokio::test]
async fn workload_discovery_stops_after_forbidden_jwks() {
let requests = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let seen = Arc::clone(&requests);
let (base, handle) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), "http://192.168.65.254:8080/jwks".into(), "/jwks".into()),
2,
"ES256",
true,
move |_| {
seen.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
serde_json::json!({"keys": []}).to_string()
},
)
.expect("workload discovery mock must bind");
let mut config = build_mocked_oidc_provider_config("workload", &base);
config.hide_from_ui = true;
let client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap());
let error = discover_provider(&config, &client)
.await
.err()
.expect("private JWKS must be blocked");
assert!(error.starts_with(OIDC_JWKS_BLOCKED_BY_OUTBOUND_POLICY), "{error}");
handle.join().unwrap();
assert_eq!(
requests.load(std::sync::atomic::Ordering::SeqCst),
1,
"a policy denial must not retry discovery"
);
}
#[tokio::test]
async fn workload_discovery_verification_and_rotation() {
for explicit_issuer in [false, true] {
let (_, initial_jwk) = oidc_es256_key_and_jwk("initial");
let (key, rotated_jwk) = oidc_es256_key_and_jwk("rotated");
let initial_jwks = serde_json::json!({"keys": [initial_jwk]}).to_string();
let rotated_jwks = serde_json::json!({"keys": [rotated_jwk]}).to_string();
let (base, handle) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()),
4,
"ES256",
true,
move |fetch| {
if fetch == 0 {
initial_jwks.clone()
} else {
rotated_jwks.clone()
}
},
)
.expect("workload discovery mock must bind");
let mut config = build_mocked_oidc_provider_config("workload", &base);
config.hide_from_ui = true;
if explicit_issuer {
config.issuer = Some(base.clone());
}
let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap());
let state = discover_provider(&config, &http_client)
.await
.expect("workload discovery must succeed");
assert!(state.metadata.authorization_endpoint().is_none());
let sys = OidcSys {
configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]),
provider_runtime: ProviderRuntime::new(http_client, HashMap::from([(config.id.clone(), state)])),
state_store: OidcStateStore::new(),
};
assert!(sys.has_providers());
assert!(sys.provider_configs().all(|provider| provider.hide_from_ui));
let error = sys
.authorize_url(&config.id, "https://console.example.com/callback", None)
.await
.unwrap_err();
assert!(error.contains("only web identity"));
let now = time::OffsetDateTime::now_utc().unix_timestamp();
let payload = serde_json::json!({
"iss": base, "sub": "system:serviceaccount:default:reader", "aud": [config.client_id],
"iat": now, "exp": now + 300, "groups": ["readonly"],
"kubernetes.io": {"namespace": "default", "serviceaccount": {"name": "reader"}},
});
let mut header = Header::new(Algorithm::ES256);
header.kid = Some("rotated".into());
let token = jsonwebtoken::encode(&header, &payload, &key).unwrap();
let (claims, provider) = sys
.verify_web_identity_token(&token)
.await
.expect("rotation must retain workload discovery support");
assert_eq!(provider, config.id);
assert_eq!(claims.sub, "system:serviceaccount:default:reader");
assert_eq!(claims.groups, ["readonly"]);
// Repeat after the mock exits: the verified snapshot must be cached.
handle.join().unwrap();
assert!(sys.verify_web_identity_token(&token).await.is_ok());
for (field, value, expected) in [
("iss", serde_json::json!("https://wrong.example.com"), "issuer"),
("aud", serde_json::json!("wrong-audience"), "audience"),
("exp", serde_json::json!(now - 60), "expired"),
] {
let mut invalid = payload.clone();
invalid[field] = value;
let token = jsonwebtoken::encode(&header, &invalid, &key).unwrap();
let error = sys
.verify_web_identity_token(&token)
.await
.expect_err("invalid workload token must fail");
assert!(error.to_lowercase().contains(expected), "{field}: {error}");
}
let (wrong_key, _) = oidc_es256_key_and_jwk("wrong");
let invalid = jsonwebtoken::encode(&header, &payload, &wrong_key).unwrap();
let error = sys
.verify_web_identity_token(&invalid)
.await
.expect_err("wrong signature must fail");
assert!(error.to_lowercase().contains("signature"), "{error}");
assert!(
sys.verify_web_identity_token(&token).await.is_ok(),
"failed refresh must preserve the cached keys"
);
}
}
#[tokio::test]
async fn oidc_discovery_accepts_extra_root_ca_for_https_provider() {
let Some((base, ca_pem, handle)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
return;
};
let config_url = format!("{base}/application/o/rustfs");
let config = build_mocked_oidc_provider_config("default", &config_url);
let origin = Url::parse(&config.config_url)
.expect("mock config_url should parse")
.origin()
.ascii_serialization();
let policy = OutboundPolicy::from_allowed_origins(&origin).expect("loopback TLS origin should be allowed");
let extra_root_certs =
parse_oidc_extra_root_certs("test OIDC TLS CA", ca_pem.as_bytes()).expect("test CA bundle should parse");
let http_client = ReqwestHttpClient::with_policy_and_extra_root_certs(policy, extra_root_certs);
let state = discover_provider(&config, &http_client)
.await
.expect("OIDC discovery should trust the extra root CA");
assert_eq!(state.metadata.issuer().to_string(), format!("{base}/application/o/rustfs"));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn oidc_discovery_refreshes_extra_root_ca_when_generation_changes() {
let Some((base_a, ca_pem_a, handle_a)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs-a"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
return;
};
let Some((base_b, ca_pem_b, handle_b)) = start_mock_oidc_tls_discovery_server(
|base| (format!("{base}/application/o/rustfs-b"), format!("{base}/jwks"), "/jwks".to_string()),
4,
) else {
assert!(handle_a.join().is_ok());
return;
};
let origin_a = Url::parse(&base_a)
.expect("mock base A should parse")
.origin()
.ascii_serialization();
let origin_b = Url::parse(&base_b)
.expect("mock base B should parse")
.origin()
.ascii_serialization();
let allowed_origins = format!("{origin_a},{origin_b}");
let policy = OutboundPolicy::from_allowed_origins(&allowed_origins).expect("loopback TLS origins should be allowed");
let material = Arc::new(Mutex::new(OidcExtraRootCaMaterial {
generation: 1,
root_ca_pem: Some(ca_pem_a.into_bytes()),
}));
let provider = OidcExtraRootCaProvider::new({
let material = material.clone();
move || {
let material = material.clone();
async move {
material
.lock()
.map(|material| material.clone())
.map_err(|e| format!("test OIDC extra CA material lock poisoned: {e}"))
}
}
});
let http_client = ReqwestHttpClient::with_policy_and_extra_root_ca_provider(policy, provider);
let config_a = build_mocked_oidc_provider_config("a", &format!("{base_a}/application/o/rustfs-a"));
let state_a = discover_provider(&config_a, &http_client)
.await
.expect("OIDC discovery should trust initial extra root CA");
assert_eq!(state_a.metadata.issuer().to_string(), format!("{base_a}/application/o/rustfs-a"));
{
let mut material = material
.lock()
.expect("test OIDC extra CA material lock should not be poisoned");
material.generation = 2;
material.root_ca_pem = Some(ca_pem_b.into_bytes());
}
let config_b = build_mocked_oidc_provider_config("b", &format!("{base_b}/application/o/rustfs-b"));
let state_b = discover_provider(&config_b, &http_client)
.await
.expect("OIDC discovery should refresh extra root CA after generation change");
assert_eq!(state_b.metadata.issuer().to_string(), format!("{base_b}/application/o/rustfs-b"));
assert!(handle_a.join().is_ok());
assert!(handle_b.join().is_ok());
}
#[test]
fn test_jwks_url_from_config_url_preserves_unrelated_urls() {
let issuer = IssuerUrl::new("https://public.example.com/realms/app".to_string()).expect("issuer URL should parse");
for raw_jwks_url in [
"https://keys.example.com/jwks",
"http://public.example.com/realms/app/jwks",
"https://public.example.com:8443/realms/app/jwks",
"https://public.example.com/keys/jwks",
"https://public.example.com/realms/application/jwks",
] {
let jwks_url = JsonWebKeySetUrl::new(raw_jwks_url.to_string()).expect("JWKS URL should parse");
let resolved =
jwks_url_from_config_url("http://keycloak.internal/realms/app/.well-known/openid-configuration", &issuer, &jwks_url)
.expect("JWKS URL should resolve");
assert_eq!(resolved.as_str(), raw_jwks_url);
}
let issuer_root_jwks = JsonWebKeySetUrl::new(issuer.as_str().to_string()).expect("JWKS URL should parse");
let resolved = jwks_url_from_config_url(
"http://keycloak.internal/realms/app/.well-known/openid-configuration",
&issuer,
&issuer_root_jwks,
)
.expect("issuer-root JWKS URL should resolve");
assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app");
let issuer_with_slash =
IssuerUrl::new("https://public.example.com/realms/app/".to_string()).expect("issuer URL should parse");
let jwks_url =
JsonWebKeySetUrl::new("https://public.example.com/realms/app/jwks".to_string()).expect("JWKS URL should parse");
let resolved = jwks_url_from_config_url(
"http://keycloak.internal/realms/app/.well-known/openid-configuration",
&issuer_with_slash,
&jwks_url,
)
.expect("issuer-relative JWKS URL should resolve");
assert_eq!(resolved.as_str(), "http://keycloak.internal/realms/app/jwks");
}
#[tokio::test]
async fn oidc_discovery_reports_forbidden_outbound_without_retrying() {
let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration";
let config = build_mocked_oidc_provider_config("default", config_url);
let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default());
let error = match discover_provider(&config, &http_client).await {
Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"),
Err(error) => error,
};
assert!(error.contains("OIDC provider discovery blocked by outbound policy"));
assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
assert!(!error.contains("discovery failed for all issuer variants"));
}
#[tokio::test]
async fn oidc_explicit_issuer_reports_forbidden_discovery_endpoint() {
let config_url = "http://192.168.65.254:8080/realms/rustfs/.well-known/openid-configuration";
let mut config = build_mocked_oidc_provider_config("default", config_url);
config.issuer = Some("https://idp.example.com/realms/rustfs".to_string());
let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::default());
let error = match discover_provider(&config, &http_client).await {
Ok(_) => panic!("private OIDC provider should require an explicit allowlist origin"),
Err(error) => error,
};
assert!(error.contains("OIDC provider discovery blocked by outbound policy"));
assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
}
#[tokio::test]
async fn oidc_explicit_issuer_reports_forbidden_jwks_endpoint() {
let Some((base, handle)) = start_mock_oidc_discovery_server(
|base| {
(
format!("{base}/realms/rustfs"),
"http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/certs".to_string(),
"/unused".to_string(),
)
},
1,
) else {
return;
};
let mut config =
build_mocked_oidc_provider_config("default", &format!("{base}/realms/rustfs/.well-known/openid-configuration"));
config.issuer = Some(format!("{base}/realms/rustfs"));
let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback discovery origin should be allowed");
let http_client = ReqwestHttpClient::with_policy(policy);
let error = match discover_provider(&config, &http_client).await {
Ok(_) => panic!("private JWKS endpoint should require an explicit allowlist origin"),
Err(error) => error,
};
assert!(error.contains("JWKS request blocked by outbound policy"));
assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn oidc_reqwest_dns_policy_rejection_stays_typed() {
let calls = Arc::new(AtomicUsize::new(0));
let config = build_mocked_oidc_provider_config(
"default",
"http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration",
);
let http_client = ReqwestHttpClient::with_policy_and_dns_resolver(
OutboundPolicy::default(),
Arc::new(RejectingDnsResolver {
allow_origin_can_recover: true,
calls: Some(calls.clone()),
}),
);
let error = match discover_provider(&config, &http_client).await {
Ok(_) => panic!("private DNS answer should fail discovery"),
Err(error) => error,
};
assert!(error.contains("OIDC provider discovery blocked by outbound policy"));
assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried");
}
#[tokio::test]
async fn oidc_explicit_issuer_preserves_dns_policy_rejection() {
let calls = Arc::new(AtomicUsize::new(0));
let mut config = build_mocked_oidc_provider_config(
"default",
"http://keycloak.internal:8080/realms/rustfs/.well-known/openid-configuration",
);
config.issuer = Some("https://idp.example.com/realms/rustfs".to_string());
let http_client = ReqwestHttpClient::with_policy_and_dns_resolver(
OutboundPolicy::default(),
Arc::new(RejectingDnsResolver {
allow_origin_can_recover: true,
calls: Some(calls.clone()),
}),
);
let error = match discover_provider(&config, &http_client).await {
Ok(_) => panic!("private DNS answer should fail discovery"),
Err(error) => error,
};
assert!(error.contains("OIDC provider discovery blocked by outbound policy"));
assert!(error.contains(&format!("add http://keycloak.internal:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
assert_eq!(calls.load(Ordering::Relaxed), 1, "policy rejection must not be retried");
}
#[tokio::test]
async fn workload_config_validation_reports_absent_console_endpoints() {
let (base, handle) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), format!("{base}/jwks"), "/jwks".into()),
2,
"ES256",
true,
|_| serde_json::json!({"keys": []}).to_string(),
)
.expect("workload discovery mock must bind");
let mut config = build_mocked_oidc_provider_config("workload", &base);
config.hide_from_ui = true;
// Inferred issuers keep the library's discovery URL construction.
config.config_url = format!("{base}/.well-known/openid-configuration?ignored=1#ignored");
let result = validate_mocked_oidc_provider_config(&config)
.await
.expect("hidden workload configuration should validate");
assert_eq!(result.issuer, base);
assert!(result.authorization_endpoint.is_none());
assert!(result.token_endpoint.is_none());
handle.join().unwrap();
}
#[tokio::test]
async fn test_validate_oidc_provider_config_retries_with_issuer_candidates() {
// Discovery document must advertise the canonical issuer path. The first candidate has no
// trailing slash; openidconnect rejects issuer mismatch, then the second variant succeeds.
let Some((base, handle)) = start_mock_oidc_discovery_server(
|base| (format!("{base}/application/o/rustfs/"), format!("{base}/jwks"), "/jwks".to_string()),
8,
) else {
return;
};
let config_url = format!("{base}/application/o/rustfs");
let config = build_mocked_oidc_provider_config("default", &config_url);
let result = validate_mocked_oidc_provider_config(&config).await;
let validation_result = result.expect("OIDC provider validation should succeed");
assert_eq!(validation_result.issuer, format!("{base}/application/o/rustfs/"));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn test_validate_oidc_provider_config_fetches_issuer_relative_jwks_from_config_url() {
let Some((base, handle)) = start_mock_oidc_discovery_server(
|_| {
(
"http://127.0.0.1:1/public/realms/app".to_string(),
"http://127.0.0.1:1/public/realms/app/jwks?version=1".to_string(),
"/internal/realms/app/jwks?version=1".to_string(),
)
},
2,
) else {
return;
};
let mut config =
build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration"));
config.issuer = Some("http://127.0.0.1:1/public/realms/app".to_string());
let validation_result = validate_mocked_oidc_provider_config(&config)
.await
.expect("OIDC provider validation should succeed");
assert_eq!(validation_result.issuer, "http://127.0.0.1:1/public/realms/app");
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn test_validate_oidc_provider_config_rejects_separate_issuer_mismatch() {
let Some((base, handle)) = start_mock_oidc_discovery_server(
|base| {
(
"https://public.example.com/realms/other".to_string(),
format!("{base}/jwks"),
"/jwks".to_string(),
)
},
1,
) else {
return;
};
let mut config =
build_mocked_oidc_provider_config("default", &format!("{base}/internal/realms/app/.well-known/openid-configuration"));
config.issuer = Some("https://public.example.com/realms/app".to_string());
let err = validate_mocked_oidc_provider_config(&config)
.await
.expect_err("OIDC provider validation should fail");
assert!(err.contains("unexpected issuer URI"));
assert!(err.contains("https://public.example.com/realms/app"));
assert!(handle.join().is_ok());
}
#[tokio::test]
async fn test_validate_oidc_provider_config_returns_detailed_errors() {
let Some((base, handle)) = start_mock_oidc_discovery_server(
|base| (format!("{base}/application/o/other"), format!("{base}/jwks"), "/jwks".to_string()),
8,
) else {
return;
};
let config_url = format!("{base}/application/o/rustfs");
let config = build_mocked_oidc_provider_config("default", &config_url);
let err = validate_mocked_oidc_provider_config(&config)
.await
.expect_err("OIDC provider validation should fail");
assert!(discovery_error_contains_all_variants(&err, &base));
assert!(err.contains("issuer '"));
assert!(err.contains(&format!("issuer '{base}/application/o/rustfs'")));
assert!(err.contains(&format!("issuer '{base}/application/o/rustfs/'")));
assert!(handle.join().is_ok());
}
crates/iam/src/oidc/runtime.rs
+897
View File
@@ -0,0 +1,897 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::{
config::{
OidcConfigQuery, OidcConfigSnapshot, OidcProviderConfig, OidcProviderSummary, OidcSiteReplicationSnapshot,
SourcedOidcProviderConfig, load_oidc_config_snapshot,
},
provider::{ProviderRuntime, discover_provider},
state::{OidcAuthSession, OidcLogoutSession, OidcStateStore},
transport::{OidcExtraRootCaProvider, ReqwestHttpClient, inspect_token_response_body, oidc_http_error_diagnostics},
};
#[cfg(test)]
use crate::federation::FederatedAuthorizationRuleRef;
use crate::{
EVENT_OIDC_DIAGNOSTICS, LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC,
federation::{
CoreFederatedAuthorizationMapper, FederatedAuthorizationRule, FederatedAuthorizationRules, FederatedClaims,
FederatedIdentityService, FederatedProviderQuery, FederatedProviderRef, FederatedProviderView, FederatedRedirectPolicy,
FederationError, OpaqueLogoutContinuation, Result as FederationResult, StandardOidcAuthentication,
VerifiedFederatedCodeExchange, VerifiedFederatedIdentity,
},
};
use openidconnect::core::{CoreAuthenticationFlow, CoreClient, CoreIdToken};
use openidconnect::{
Audience, AuthType, AuthorizationCode, ClientId, ClientSecret, CsrfToken, LogoutRequest, Nonce, PkceCodeChallenge,
PkceCodeVerifier, PostLogoutRedirectUrl, RedirectUrl, RequestTokenError, Scope,
};
use rustfs_policy::policy::{ClaimLookup, get_claim_case_insensitive};
use serde::{Deserialize, Serialize};
use std::borrow::Cow;
use std::collections::HashMap;
use std::sync::Arc;
use tracing::{debug, error, warn};
/// Claims extracted from an OIDC ID token.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
pub(crate) struct OidcClaims {
pub sub: String,
pub email: String,
pub username: String,
pub groups: Vec<String>,
pub raw: HashMap<String, serde_json::Value>,
}
// ---- Core OIDC system ----
/// Global OIDC manager for all configured providers.
pub struct OidcSys {
pub(super) configs: HashMap<String, SourcedOidcProviderConfig>,
pub(super) provider_runtime: ProviderRuntime,
pub(super) state_store: OidcStateStore,
}
pub(super) fn trusted_aud(other_audiences: &[String], audience: &Audience) -> bool {
for aud in other_audiences {
if audience.as_str() == aud.as_str() {
return true;
}
}
false
}
impl OidcSys {
pub(crate) async fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<Self, String> {
let http_client = ReqwestHttpClient::new_with_extra_root_ca_provider(extra_root_ca_provider)?;
http_client.current_extra_root_certs().await.map_err(|err| err.to_string())?;
Self::new_with_http_client(http_client).await
}
async fn new_with_http_client(http_client: ReqwestHttpClient) -> Result<Self, String> {
let server_config = crate::server_config::current_server_config();
let parsed_configs = load_oidc_config_snapshot(server_config.as_ref());
let mut configs = HashMap::new();
let mut provider_states = HashMap::new();
for sourced_config in parsed_configs.into_providers() {
let config = &sourced_config.config;
if !config.enabled {
debug!(provider = %config.id, "OIDC provider disabled");
continue;
}
match discover_provider(config, &http_client).await {
Ok(state) => {
debug!(provider = %config.id, "OIDC provider discovered");
provider_states.insert(config.id.clone(), state);
configs.insert(config.id.clone(), sourced_config);
}
Err(e) => {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "provider_discovery_failed",
provider_id = %config.id,
config_url = %config.config_url,
client_id = %config.client_id,
scopes = ?config.scopes,
redirect_uri = %config.redirect_uri.as_deref().unwrap_or(""),
redirect_uri_dynamic = config.redirect_uri_dynamic,
error = %e,
"oidc provider discovery failed"
);
}
}
}
Ok(Self {
configs,
provider_runtime: ProviderRuntime::new(http_client, provider_states),
state_store: OidcStateStore::new(),
})
}
/// Create an OidcSys with no providers (useful for when OIDC is not configured).
pub fn empty() -> Result<Self, String> {
Ok(Self {
configs: HashMap::new(),
provider_runtime: ProviderRuntime::new(ReqwestHttpClient::new()?, HashMap::new()),
state_store: OidcStateStore::new(),
})
}
/// Return true if any OIDC providers are configured and enabled.
pub(crate) fn has_providers(&self) -> bool {
!self.configs.is_empty()
}
pub(crate) fn provider_configs(&self) -> impl Iterator<Item = &OidcProviderConfig> {
self.configs.values().map(|provider| &provider.config)
}
/// List all providers, including providers hidden from the login UI.
pub(crate) fn list_providers(&self) -> Vec<OidcProviderSummary> {
self.provider_configs()
.map(|config| OidcProviderSummary {
provider_id: config.id.clone(),
display_name: config.display_name.clone(),
})
.collect()
}
/// List providers visible in the login UI.
pub(crate) fn list_visible_providers(&self) -> Vec<OidcProviderSummary> {
self.provider_configs()
.filter(|config| !config.hide_from_ui)
.map(|config| OidcProviderSummary {
provider_id: config.id.clone(),
display_name: config.display_name.clone(),
})
.collect()
}
pub(crate) fn config_snapshot(&self) -> OidcConfigSnapshot {
OidcConfigSnapshot::new(self.configs.values().cloned().collect())
}
/// Build the PKCE authorization URL for a provider, store state in the state store.
pub(crate) async fn authorize_url(
&self,
provider_id: &str,
redirect_uri: &str,
redirect_after: Option<String>,
) -> Result<String, String> {
let config = self
.get_provider_config(provider_id)
.ok_or_else(|| format!("unknown OIDC provider: {provider_id}"))?;
let state = self.provider_runtime.ensure_provider_state(provider_id, config).await?;
let (pkce_challenge, pkce_verifier) = PkceCodeChallenge::new_random_sha256();
let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?;
let client = CoreClient::from_provider_metadata(
state.metadata.console()?.clone(),
ClientId::new(config.client_id.clone()),
config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())),
)
.set_auth_type(AuthType::RequestBody);
let mut auth_req =
client.authorize_url(CoreAuthenticationFlow::AuthorizationCode, CsrfToken::new_random, Nonce::new_random);
auth_req = auth_req.set_redirect_uri(Cow::Owned(redirect));
for scope in &config.scopes {
auth_req = auth_req.add_scope(Scope::new(scope.clone()));
}
auth_req = auth_req.set_pkce_challenge(pkce_challenge);
let (auth_url, csrf_token, nonce) = auth_req.url();
// Store the state for callback validation
self.state_store
.insert(
csrf_token.secret().clone(),
OidcAuthSession {
provider_id: provider_id.to_string(),
pkce_verifier: pkce_verifier.secret().clone(),
nonce: nonce.secret().clone(),
redirect_after,
},
)
.await;
Ok(auth_url.to_string())
}
/// Exchange an authorization code for tokens and extract claims.
pub(super) async fn exchange_code(
&self,
state: &str,
code: &str,
redirect_uri: &str,
) -> Result<(OidcClaims, String, OidcAuthSession, String), String> {
// Retrieve and consume the state (single-use)
let session = self
.state_store
.take(state)
.await
.ok_or_else(|| "invalid or expired OIDC state".to_string())?;
let config = self
.get_provider_config(&session.provider_id)
.ok_or_else(|| format!("unknown provider: {}", session.provider_id))?;
let provider_state = self.provider_runtime.get_provider_state(&session.provider_id)?;
let issuer = provider_state.metadata.issuer().to_string();
let token_endpoint = provider_state
.metadata
.token_endpoint()
.map(ToString::to_string)
.unwrap_or_default();
// Construct CoreClient on-the-fly with JWKS from discovery
let client = CoreClient::from_provider_metadata(
provider_state.metadata.console()?.clone(),
ClientId::new(config.client_id.clone()),
config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())),
)
.set_auth_type(AuthType::RequestBody);
let redirect = RedirectUrl::new(redirect_uri.to_string()).map_err(|e| format!("invalid redirect URI: {e}"))?;
// Exchange code for tokens
let token_response = client
.exchange_code(AuthorizationCode::new(code.to_string()))
.map_err(|e| {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_endpoint_missing",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
client_id = %config.client_id,
redirect_uri = %redirect_uri,
scopes = ?config.scopes,
error = %e,
"oidc token exchange failed"
);
format!(
"token endpoint not configured: {e}: provider_id={}, config_url={}, issuer={}, redirect_uri={}, client_id={}",
session.provider_id, config.config_url, issuer, redirect_uri, config.client_id
)
})?
.set_pkce_verifier(PkceCodeVerifier::new(session.pkce_verifier.clone()))
.set_redirect_uri(Cow::Owned(redirect))
.request_async(self.provider_runtime.http_client())
.await
.map_err(|e| match &e {
RequestTokenError::ServerResponse(response) => {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_server_response",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
client_secret_configured = config.client_secret.as_deref().is_some_and(|secret| !secret.is_empty()),
redirect_uri = %redirect_uri,
scopes = ?config.scopes,
oauth_error = %response.error(),
oauth_error_description = %response.error_description().map(String::as_str).unwrap_or(""),
oauth_error_uri = %response.error_uri().map(String::as_str).unwrap_or(""),
error = %e,
"oidc token exchange failed"
);
format!(
"token exchange failed: {e}: stage=token_server_response, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, oauth_error={}, oauth_error_description={}",
session.provider_id,
config.config_url,
issuer,
token_endpoint,
redirect_uri,
config.client_id,
response.error(),
response.error_description().map(String::as_str).unwrap_or("")
)
}
RequestTokenError::Request(err) => {
let (request_error_kind, request_error_status) = oidc_http_error_diagnostics(err);
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_request_failed",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
redirect_uri = %redirect_uri,
request_error_kind = %request_error_kind,
request_error_status = %request_error_status,
error = %err,
"oidc token exchange failed"
);
format!(
"token exchange failed: stage=token_request_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, request_error_kind={}, request_error_status={}, request_error={}",
session.provider_id,
config.config_url,
issuer,
token_endpoint,
redirect_uri,
config.client_id,
request_error_kind,
request_error_status,
err
)
}
RequestTokenError::Parse(parse_err, body) => {
let shape = inspect_token_response_body(body);
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_response_parse_failed",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
redirect_uri = %redirect_uri,
parse_error_path = %parse_err.path(),
response_body_len = body.len(),
response_json_object = shape.json_object,
response_json_keys = %shape.json_keys,
response_has_access_token = shape.has_access_token,
response_has_id_token = shape.has_id_token,
response_has_token_type = shape.has_token_type,
response_has_expires_in = shape.has_expires_in,
response_has_error = shape.has_error,
response_has_error_description = shape.has_error_description,
response_looks_like_html = shape.looks_like_html,
error = %e,
"oidc token exchange failed"
);
format!(
"token exchange failed: {e}: stage=token_response_parse_failed, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, parse_error_path={}, response_body_len={}, response_json_keys={}, response_has_id_token={}, response_has_error={}, response_looks_like_html={}",
session.provider_id,
config.config_url,
issuer,
token_endpoint,
redirect_uri,
config.client_id,
parse_err.path(),
body.len(),
shape.json_keys,
shape.has_id_token,
shape.has_error,
shape.looks_like_html
)
}
RequestTokenError::Other(message) => {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_exchange_other_error",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
redirect_uri = %redirect_uri,
error = %message,
"oidc token exchange failed"
);
format!(
"token exchange failed: {e}: stage=token_exchange_other_error, provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}",
session.provider_id, config.config_url, issuer, token_endpoint, redirect_uri, config.client_id
)
}
})?;
// Verify the ID token (signature, issuer, audience, expiry, nonce)
let id_token = token_response
.extra_fields()
.id_token()
.ok_or_else(|| {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "token_response_missing_id_token",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
redirect_uri = %redirect_uri,
scopes = ?config.scopes,
"oidc token exchange failed"
);
format!(
"no id_token in token response: provider_id={}, config_url={}, issuer={}, token_endpoint={}, redirect_uri={}, client_id={}, scopes={}",
session.provider_id,
config.config_url,
issuer,
token_endpoint,
redirect_uri,
config.client_id,
config.scopes.join(",")
)
})?;
let verifier = client
.id_token_verifier()
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
let verified = id_token.claims(&verifier, &Nonce::new(session.nonce.clone()));
if let Err(e) = verified {
let verification_error = e.to_string();
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "id_token_verification_retry",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
other_audiences = ?config.other_audiences,
error = %verification_error,
"oidc id token verification failed"
);
let refreshed_state = self
.provider_runtime
.refresh_provider_state(&session.provider_id, config)
.await
.map_err(|refresh_err| {
format!(
"ID token verification failed: {verification_error}; failed to refresh provider metadata: {refresh_err}"
)
})?;
warn!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "jwks_metadata_refreshed",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
"oidc provider metadata refreshed"
);
let client = CoreClient::from_provider_metadata(
refreshed_state.metadata.console()?.clone(),
ClientId::new(config.client_id.clone()),
config.client_secret.as_ref().map(|s| ClientSecret::new(s.clone())),
)
.set_auth_type(AuthType::RequestBody);
let verifier = client
.id_token_verifier()
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
id_token
.claims(&verifier, &Nonce::new(session.nonce.clone()))
.map_err(|retry_err| {
error!(
event = EVENT_OIDC_DIAGNOSTICS,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "id_token_verification_failed",
provider_id = %session.provider_id,
config_url = %config.config_url,
issuer = %issuer,
token_endpoint = %token_endpoint,
client_id = %config.client_id,
other_audiences = ?config.other_audiences,
original_error = %verification_error,
retry_error = %retry_err,
"oidc id token verification failed"
);
format!(
"ID token verification failed after JWKS refresh: {retry_err}; original_error={verification_error}; provider_id={}, config_url={}, issuer={}, token_endpoint={}, client_id={}, other_audiences={}",
session.provider_id,
config.config_url,
issuer,
token_endpoint,
config.client_id,
config.other_audiences.join(",")
)
})?;
}
// Extract raw claims from the verified JWT for custom claim support
// (the crate verifies signature/expiry/nonce; we decode payload for non-standard claims)
let raw_jwt = id_token.to_string();
let raw = decode_jwt_payload(&raw_jwt);
let claims = OidcClaims {
sub: extract_string_claim(&raw, "sub"),
email: extract_string_claim(&raw, &config.email_claim),
username: extract_string_claim(&raw, &config.username_claim),
groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim),
raw,
};
Ok((claims, session.provider_id.clone(), session, raw_jwt))
}
/// Store a one-time logout session keyed by an opaque token so the console can
/// trigger browser logout without persisting the raw ID token.
pub(crate) async fn create_logout_token(&self, provider_id: &str, id_token: &str) -> Result<String, String> {
if !self.configs.contains_key(provider_id) {
return Err(format!("unknown OIDC provider: {provider_id}"));
}
let token = CsrfToken::new_random().secret().clone();
self.state_store
.insert_logout(
token.clone(),
OidcLogoutSession {
provider_id: provider_id.to_string(),
id_token: id_token.to_string(),
},
)
.await;
Ok(token)
}
/// Build the RP-initiated logout URL for a previously issued logout token.
/// Returns `Ok(None)` when the provider does not advertise an end-session endpoint.
pub(crate) async fn build_logout_url(
&self,
logout_token: &str,
post_logout_redirect_uri: &str,
) -> Result<Option<String>, String> {
let session = self
.state_store
.take_logout(logout_token)
.await
.ok_or_else(|| "invalid or expired OIDC logout token".to_string())?;
let config = self
.get_provider_config(&session.provider_id)
.ok_or_else(|| format!("unknown OIDC provider: {}", session.provider_id))?;
let state = self
.provider_runtime
.ensure_provider_state(&session.provider_id, config)
.await?;
let Some(end_session_endpoint) = state.metadata.console()?.additional_metadata().end_session_endpoint.clone() else {
return Ok(None);
};
let id_token: CoreIdToken = session
.id_token
.parse()
.map_err(|e: serde_json::Error| format!("failed to parse ID token for logout: {e}"))?;
let post_logout_redirect_uri = PostLogoutRedirectUrl::new(post_logout_redirect_uri.to_string())
.map_err(|e| format!("invalid post logout redirect URI: {e}"))?;
let logout_url = LogoutRequest::from(end_session_endpoint)
.set_id_token_hint(&id_token)
.set_client_id(ClientId::new(config.client_id.clone()))
.set_post_logout_redirect_uri(post_logout_redirect_uri)
.http_get_url()
.to_string();
Ok(Some(logout_url))
}
/// Map OIDC claims to rustfs policy names.
#[cfg(test)]
pub(crate) fn map_claims_to_policies(&self, provider_id: &str, claims: &OidcClaims) -> (Vec<String>, Vec<String>) {
let Some(rule) = self.authorization_rule(provider_id) else {
return (Vec::new(), Vec::new());
};
CoreFederatedAuthorizationMapper::map_policies_and_groups(provider_id, rule, &claims.groups, &claims.raw, true)
}
/// Policy names produced only by the canonical groups in claim-based mode: the groups claim
/// plus the roles claim values that `extract_canonical_group_values` merges into it.
///
/// Identity providers emit built-in groups and roles that can never have a matching policy
/// (for example `DOMAIN\Domain Users` or Keycloak's `offline_access`), so the session binding
/// may ignore these names when no such policy exists. Names from a fixed role policy or a
/// dedicated policy claim are never included, so those configurations keep requiring every
/// policy to resolve.
#[cfg(test)]
pub(crate) fn group_claim_policy_names(&self, provider_id: &str, claims: &OidcClaims) -> Vec<String> {
let Some(rule) = self.authorization_rule(provider_id) else {
return Vec::new();
};
CoreFederatedAuthorizationMapper::group_claim_policy_names(rule, &claims.groups, &claims.raw)
}
#[cfg(test)]
fn authorization_rule(&self, provider_id: &str) -> Option<FederatedAuthorizationRuleRef<'_>> {
self.get_provider_config(provider_id).map(|config| {
FederatedAuthorizationRuleRef::new(
&config.claim_name,
&config.claim_prefix,
&config.role_policy,
&config.groups_claim,
&config.roles_claim,
)
})
}
/// Verify a raw JWT (id_token) for the AssumeRoleWithWebIdentity flow.
///
/// Unlike the authorization code flow, ARWWI receives a raw JWT directly
/// (not via code exchange). This method:
/// 1. Decodes the JWT payload to extract the `iss` claim
/// 2. Finds the OIDC provider whose issuer matches
/// 3. Verifies signature, issuer, audience, and expiry (nonce is skipped)
/// 4. Extracts claims using the provider's claim configuration
pub(crate) async fn verify_web_identity_token(&self, jwt: &str) -> Result<(OidcClaims, String /* provider_id */), String> {
// Decode JWT payload without verification to get the issuer claim
let raw_claims = decode_jwt_payload(jwt);
let issuer = raw_claims
.get("iss")
.and_then(|v| v.as_str())
.ok_or_else(|| "JWT missing 'iss' claim".to_string())?;
// Find matching provider by issuer
let (provider_id, config, mut state) = self
.provider_runtime
.find_provider_by_issuer(issuer, &self.configs)
.ok_or_else(|| format!("no OIDC provider configured for issuer: {issuer}"))?;
state = self
.provider_runtime
.ensure_provider_state_if_stale(&provider_id, &config, &state)
.await?;
// Parse raw JWT string into CoreIdToken
let id_token: CoreIdToken = jwt
.parse()
.map_err(|e: serde_json::Error| format!("failed to parse JWT as ID token: {e}"))?;
// Verify the token (signature, issuer, audience, expiry) — skip nonce
// (nonce is only required for the authorization code flow)
let verifier = state
.metadata
.verifier(&config)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
if let Err(e) = id_token.claims(&verifier, |_: Option<&Nonce>| Ok(())) {
state = self
.provider_runtime
.refresh_provider_state(&provider_id, &config)
.await
.map_err(|refresh_err| {
format!("ID token verification failed: {e}; failed to refresh provider metadata: {refresh_err}")
})?;
let verifier = state
.metadata
.verifier(&config)
.set_other_audience_verifier_fn(|aud| trusted_aud(&config.other_audiences, aud));
id_token
.claims(&verifier, |_: Option<&Nonce>| Ok(()))
.map_err(|retry_err| format!("ID token verification failed after JWKS refresh: {retry_err}"))?;
}
// Extract claims using the provider's claim configuration
let claims = OidcClaims {
sub: extract_string_claim(&raw_claims, "sub"),
email: extract_string_claim(&raw_claims, &config.email_claim),
username: extract_string_claim(&raw_claims, &config.username_claim),
groups: extract_canonical_group_values(&raw_claims, &config.groups_claim, &config.roles_claim),
raw: raw_claims,
};
Ok((claims, provider_id.to_string()))
}
/// Get a provider config by ID.
pub(crate) fn get_provider_config(&self, id: &str) -> Option<&OidcProviderConfig> {
self.configs.get(id).map(|provider| &provider.config)
}
}
/// Decode the payload section of a JWT without validation (token must already be verified).
pub(crate) fn decode_jwt_payload(token: &str) -> HashMap<String, serde_json::Value> {
let parts: Vec<&str> = token.split('.').collect();
if parts.len() < 2 {
return HashMap::new();
}
let payload_bytes = base64_simd::URL_SAFE_NO_PAD.decode_to_vec(parts[1]);
match payload_bytes {
Ok(bytes) => serde_json::from_slice(&bytes).unwrap_or_default(),
Err(_) => HashMap::new(),
}
}
/// Extract a string claim from raw claims with case-insensitive fallback.
fn extract_string_claim(claims: &HashMap<String, serde_json::Value>, key: &str) -> String {
match get_claim_case_insensitive(claims, key) {
ClaimLookup::Found(value) => value.as_str().unwrap_or_default().to_string(),
ClaimLookup::Missing | ClaimLookup::Ambiguous => String::new(),
}
}
/// Extract a groups/array claim from raw claims with case-insensitive fallback. Handles both string arrays and single strings.
fn extract_groups_claim(claims: &HashMap<String, serde_json::Value>, key: &str) -> Vec<String> {
match get_claim_case_insensitive(claims, key) {
ClaimLookup::Found(serde_json::Value::Array(arr)) => arr.iter().filter_map(|v| v.as_str().map(String::from)).collect(),
ClaimLookup::Found(serde_json::Value::String(s)) => s.split(',').map(|s| s.trim().to_string()).collect(),
_ => vec![],
}
}
fn extract_canonical_group_values(
claims: &HashMap<String, serde_json::Value>,
groups_claim: &str,
roles_claim: &str,
) -> Vec<String> {
let mut groups = extract_groups_claim(claims, groups_claim);
if !roles_claim.is_empty() && roles_claim != groups_claim {
groups.extend(extract_groups_claim(claims, roles_claim));
}
groups.retain(|g| !g.is_empty());
groups.sort();
groups.dedup();
groups
}
fn verified_identity(provider: FederatedProviderRef, claims: OidcClaims) -> VerifiedFederatedIdentity {
VerifiedFederatedIdentity::from_claims(
provider,
FederatedClaims {
sub: claims.sub,
email: claims.email,
username: claims.username,
groups: claims.groups,
raw: claims.raw,
},
)
}
fn authorization_rules(oidc: &OidcSys) -> FederatedAuthorizationRules {
FederatedAuthorizationRules::new(oidc.provider_configs().map(|config| {
FederatedAuthorizationRule::new(
config.id.clone(),
config.claim_name.clone(),
config.claim_prefix.clone(),
config.role_policy.clone(),
config.groups_claim.clone(),
config.roles_claim.clone(),
)
}))
}
pub struct StandardOidcAdapter {
oidc: Arc<OidcSys>,
authorization_rules: FederatedAuthorizationRules,
}
impl StandardOidcAdapter {
pub fn new(oidc: Arc<OidcSys>) -> Self {
let authorization_rules = authorization_rules(&oidc);
Self {
oidc,
authorization_rules,
}
}
pub fn authorization_rules(&self) -> FederatedAuthorizationRules {
self.authorization_rules.clone()
}
pub fn into_service(self: Arc<Self>, mapper: CoreFederatedAuthorizationMapper) -> FederatedIdentityService {
let authentication: Arc<dyn StandardOidcAuthentication> = self.clone();
let provider_query: Arc<dyn FederatedProviderQuery> = self;
FederatedIdentityService::from_standard_oidc_parts(provider_query, authentication, mapper)
}
}
impl OidcConfigQuery for StandardOidcAdapter {
fn site_replication_snapshot(&self) -> OidcSiteReplicationSnapshot {
OidcSiteReplicationSnapshot::from_config_snapshot(&self.oidc.config_snapshot())
}
}
#[async_trait::async_trait]
impl StandardOidcAuthentication for StandardOidcAdapter {
async fn authorize_url(
&self,
provider_id: &str,
redirect_uri: &str,
redirect_after: Option<String>,
) -> FederationResult<String> {
self.oidc
.authorize_url(provider_id, redirect_uri, redirect_after)
.await
.map_err(FederationError::Authorization)
}
async fn exchange_identity(
&self,
state: &str,
code: &str,
redirect_uri: &str,
) -> FederationResult<VerifiedFederatedCodeExchange> {
let (claims, provider_id, session, id_token) = self
.oidc
.exchange_code(state, code, redirect_uri)
.await
.map_err(FederationError::CodeExchange)?;
let provider = FederatedProviderRef::new(provider_id);
Ok(VerifiedFederatedCodeExchange::new(
verified_identity(provider.clone(), claims),
session.redirect_after,
OpaqueLogoutContinuation::new(provider, id_token),
))
}
async fn verify_identity(&self, jwt: &str) -> FederationResult<VerifiedFederatedIdentity> {
let (claims, provider_id) = self
.oidc
.verify_web_identity_token(jwt)
.await
.map_err(FederationError::TokenVerification)?;
Ok(verified_identity(FederatedProviderRef::new(provider_id), claims))
}
async fn create_logout_token(&self, continuation: OpaqueLogoutContinuation) -> FederationResult<String> {
let (provider, id_token) = continuation.into_parts();
self.oidc
.create_logout_token(provider.as_str(), &id_token)
.await
.map_err(FederationError::Logout)
}
async fn build_logout_url(&self, logout_token: &str, post_logout_redirect_uri: &str) -> FederationResult<Option<String>> {
self.oidc
.build_logout_url(logout_token, post_logout_redirect_uri)
.await
.map_err(FederationError::Logout)
}
}
impl FederatedProviderQuery for StandardOidcAdapter {
fn has_providers(&self) -> bool {
self.oidc.has_providers()
}
fn list_providers(&self) -> Vec<FederatedProviderView> {
self.oidc.list_providers()
}
fn list_visible_providers(&self) -> Vec<FederatedProviderView> {
self.oidc.list_visible_providers()
}
fn redirect_policy(&self, provider_id: &str) -> Option<FederatedRedirectPolicy> {
self.oidc
.get_provider_config(provider_id)
.map(|config| FederatedRedirectPolicy {
redirect_uri: config.redirect_uri.clone(),
allow_request_origin: config.redirect_uri_dynamic,
})
}
}
#[cfg(test)]
#[path = "runtime_tests.rs"]
mod tests;
crates/iam/src/oidc/runtime_tests.rs
+754
View File
@@ -0,0 +1,754 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::oidc::{config::*, provider::*, test_support::*};
use jsonwebtoken::{Algorithm, Header};
use openidconnect::ProviderMetadataWithLogout;
use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy};
use std::time::{Duration as StdDuration, Instant};
use url::Url;
#[test]
fn test_extract_string_claim() {
let mut claims = HashMap::new();
claims.insert("email".to_string(), serde_json::json!("user@example.com"));
claims.insert("sub".to_string(), serde_json::json!("12345"));
assert_eq!(extract_string_claim(&claims, "email"), "user@example.com");
assert_eq!(extract_string_claim(&claims, "sub"), "12345");
assert_eq!(extract_string_claim(&claims, "missing"), "");
}
#[test]
fn test_extract_groups_claim_array() {
let mut claims = HashMap::new();
claims.insert("groups".to_string(), serde_json::json!(["admin", "developers", "readonly"]));
let groups = extract_groups_claim(&claims, "groups");
assert_eq!(groups, vec!["admin", "developers", "readonly"]);
}
#[test]
fn test_extract_groups_claim_string() {
let mut claims = HashMap::new();
claims.insert("groups".to_string(), serde_json::json!("admin,developers"));
let groups = extract_groups_claim(&claims, "groups");
assert_eq!(groups, vec!["admin", "developers"]);
}
#[test]
fn test_extract_groups_claim_missing() {
let claims = HashMap::new();
let groups = extract_groups_claim(&claims, "groups");
assert!(groups.is_empty());
}
#[test]
fn test_extract_groups_claim_number() {
let mut claims = HashMap::new();
claims.insert("groups".to_string(), serde_json::json!(42));
let groups = extract_groups_claim(&claims, "groups");
assert!(groups.is_empty());
}
#[test]
fn test_extract_canonical_group_values_merges_groups_and_roles() {
let mut claims = HashMap::new();
claims.insert("groups".to_string(), serde_json::json!(["devs", "admins"]));
claims.insert("roles".to_string(), serde_json::json!(["admins", "consoleAdmin"]));
let merged = extract_canonical_group_values(&claims, "groups", "roles");
assert_eq!(merged, vec!["admins", "consoleAdmin", "devs"]);
}
#[test]
fn test_extract_canonical_group_values_skips_duplicate_claim_name() {
let mut claims = HashMap::new();
claims.insert("roles".to_string(), serde_json::json!(["consoleAdmin"]));
let merged = extract_canonical_group_values(&claims, "roles", "roles");
assert_eq!(merged, vec!["consoleAdmin"]);
}
#[test]
fn test_extract_canonical_group_values_roles_only() {
let mut claims = HashMap::new();
claims.insert("roles".to_string(), serde_json::json!(["consoleAdmin", "bucket-reader"]));
let merged = extract_canonical_group_values(&claims, "groups", "roles");
assert_eq!(merged, vec!["bucket-reader", "consoleAdmin"]);
}
#[test]
fn test_extract_string_claim_case_insensitive() {
let mut claims = HashMap::new();
claims.insert("policyminio".to_string(), serde_json::json!("consoleAdmin"));
assert_eq!(extract_string_claim(&claims, "policyMinio"), "consoleAdmin");
assert_eq!(extract_string_claim(&claims, "POLICYMINIO"), "consoleAdmin");
assert_eq!(extract_string_claim(&claims, "policyminio"), "consoleAdmin");
}
#[test]
fn test_extract_groups_claim_case_insensitive() {
let mut claims = HashMap::new();
claims.insert("policyminio".to_string(), serde_json::json!(["consoleAdmin", "readwrite"]));
let groups = extract_groups_claim(&claims, "policyMinio");
assert_eq!(groups, vec!["consoleAdmin", "readwrite"]);
let groups = extract_groups_claim(&claims, "POLICYMINIO");
assert_eq!(groups, vec!["consoleAdmin", "readwrite"]);
let groups = extract_groups_claim(&claims, "policyminio");
assert_eq!(groups, vec!["consoleAdmin", "readwrite"]);
}
#[test]
fn test_extract_groups_claim_exact_match_preferred() {
let mut claims = HashMap::new();
claims.insert("Policy".to_string(), serde_json::json!(["exact_match"]));
claims.insert("policy".to_string(), serde_json::json!(["lowercase"]));
let groups = extract_groups_claim(&claims, "Policy");
assert_eq!(groups, vec!["exact_match"]);
}
#[test]
fn test_extract_string_claim_ambiguous_case_insensitive_match_returns_empty() {
let mut claims = HashMap::new();
claims.insert("Policy".to_string(), serde_json::json!("exact_match"));
claims.insert("policy".to_string(), serde_json::json!("lowercase"));
assert_eq!(extract_string_claim(&claims, "POLICY"), "");
}
#[test]
fn test_extract_groups_claim_ambiguous_case_insensitive_match_returns_empty() {
let mut claims = HashMap::new();
claims.insert("Policy".to_string(), serde_json::json!(["exact_match"]));
claims.insert("policy".to_string(), serde_json::json!(["lowercase"]));
let groups = extract_groups_claim(&claims, "POLICY");
assert!(groups.is_empty());
}
#[test]
fn test_decode_jwt_payload() {
let payload = r#"{"sub":"user123","email":"user@example.com"}"#;
let payload_b64 = base64_simd::URL_SAFE_NO_PAD.encode_to_string(payload.as_bytes());
let token = format!("eyJhbGciOiJSUzI1NiJ9.{payload_b64}.signature");
let claims = decode_jwt_payload(&token);
assert_eq!(claims.get("sub").and_then(|v| v.as_str()), Some("user123"));
assert_eq!(claims.get("email").and_then(|v| v.as_str()), Some("user@example.com"));
}
#[tokio::test]
async fn web_identity_verification_refreshes_rotated_jwks() {
let (_, initial_jwk) = oidc_es256_key_and_jwk("initial");
let (rotated_key, rotated_jwk) = oidc_es256_key_and_jwk("rotated");
let initial_jwks = serde_json::json!({ "keys": [initial_jwk] }).to_string();
let rotated_jwks = serde_json::json!({ "keys": [rotated_jwk] }).to_string();
let Some((base, handle)) = start_mock_oidc_discovery_server_with_jwks(
|base| (base.to_string(), format!("{base}/jwks"), "/jwks".to_string()),
4,
"ES256",
false,
move |fetch| {
if fetch == 0 {
initial_jwks.clone()
} else {
rotated_jwks.clone()
}
},
) else {
return;
};
let config = build_mocked_oidc_provider_config("rotating", &base);
let policy = OutboundPolicy::from_allowed_origins(&base).expect("loopback origin should be allowed");
let http_client = ReqwestHttpClient::with_policy(policy);
let state = discover_provider(&config, &http_client)
.await
.expect("initial OIDC discovery should succeed");
let sys = OidcSys {
configs: HashMap::from([(config.id.clone(), test_sourced_config(config.clone()))]),
provider_runtime: ProviderRuntime::new(http_client, HashMap::from([(config.id.clone(), state)])),
state_store: OidcStateStore::new(),
};
let now = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.expect("system clock should be after Unix epoch")
.as_secs();
let mut header = Header::new(Algorithm::ES256);
header.kid = Some("rotated".to_string());
let token = jsonwebtoken::encode(
&header,
&serde_json::json!({
"iss": base,
"sub": "rotated-user",
"aud": config.client_id,
"iat": now,
"exp": now + 300,
"groups": ["readwrite"],
}),
&rotated_key,
)
.expect("rotated OIDC token should sign");
let (claims, provider_id) = sys
.verify_web_identity_token(&token)
.await
.expect("verification should refresh JWKS and accept the rotated key");
assert_eq!(provider_id, "rotating");
assert_eq!(claims.sub, "rotated-user");
assert_eq!(claims.groups, vec!["readwrite"]);
handle.join().expect("rotating JWKS mock server should exit cleanly");
}
#[tokio::test]
async fn complete_provider_console_login_preserves_hidden_and_issuer_modes() {
use tokio::io::{AsyncReadExt, AsyncWriteExt};
for hidden in [false, true] {
for explicit in [false, true] {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap();
let base = format!("http://{}", listener.local_addr().unwrap());
let issuer = if explicit {
"https://issuer.example.com".to_string()
} else {
base.clone()
};
let mut config = build_mocked_oidc_provider_config("console", &format!("{base}/.well-known/openid-configuration"));
config.hide_from_ui = hidden;
config.issuer = explicit.then(|| issuer.clone());
config.client_secret = Some(Nonce::new_random().secret().clone());
let server_config = config.clone();
let server_base = base.clone();
let redirect = "https://console.example.com/oauth_callback";
let (key, jwk) = oidc_es256_key_and_jwk("console");
let (auth_tx, auth_rx) = tokio::sync::oneshot::channel::<HashMap<String, String>>();
let server = tokio::spawn(async move {
let mut auth_rx = Some(auth_rx);
for expected_path in ["/.well-known/openid-configuration", "/jwks", "/token"] {
let (mut stream, _) = tokio::time::timeout(StdDuration::from_secs(10), listener.accept())
.await
.unwrap()
.unwrap();
let mut bytes = Vec::new();
let header_end = loop {
bytes.push(stream.read_u8().await.unwrap());
assert!(bytes.len() < 8192);
if bytes.ends_with(b"\r\n\r\n") {
break bytes.len();
}
};
let headers = String::from_utf8(bytes).unwrap();
let request_line = headers.lines().next().unwrap();
assert_eq!(request_line.split_whitespace().nth(1), Some(expected_path));
let headers_map: HashMap<_, _> = headers
.lines()
.skip(1)
.filter_map(|line| line.split_once(':'))
.map(|(name, value)| (name.to_ascii_lowercase(), value.trim().to_string()))
.collect();
let body = match expected_path {
"/.well-known/openid-configuration" => {
assert!(request_line.starts_with("GET "));
assert_eq!(headers_map["accept"], "application/json");
serde_json::json!({
"issuer": issuer, "authorization_endpoint": format!("{server_base}/authorize"),
"token_endpoint": format!("{server_base}/token"), "jwks_uri": format!("{server_base}/jwks"),
"response_types_supported": ["code"], "subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["ES256"]
})
}
"/jwks" => {
assert!(request_line.starts_with("GET "));
assert!(headers_map["accept"].contains("application/json"));
serde_json::json!({"keys": [jwk]})
}
"/token" => {
assert!(request_line.starts_with("POST "));
assert_eq!(headers_map["accept"], "application/json");
assert!(headers_map["content-type"].starts_with("application/x-www-form-urlencoded"));
let length: usize = headers_map["content-length"].parse().unwrap();
assert!(header_end + length < 16384);
let mut body = vec![0; length];
stream.read_exact(&mut body).await.unwrap();
let form: HashMap<String, String> = url::form_urlencoded::parse(&body).into_owned().collect();
assert_eq!(form["grant_type"], "authorization_code");
assert_eq!(form["code"], "test-authorization-code");
assert_eq!(form["client_id"], server_config.client_id);
assert_eq!(Some(&form["client_secret"]), server_config.client_secret.as_ref());
assert_eq!(form["redirect_uri"], redirect);
let auth = auth_rx.take().unwrap().await.unwrap();
let challenge = PkceCodeChallenge::from_code_verifier_sha256(&PkceCodeVerifier::new(
form["code_verifier"].clone(),
));
assert_eq!(challenge.as_str(), auth["code_challenge"]);
let now = time::OffsetDateTime::now_utc().unix_timestamp();
let mut header = Header::new(Algorithm::ES256);
header.kid = Some("console".into());
let token = jsonwebtoken::encode(
&header,
&serde_json::json!({
"iss": issuer, "sub": "existing-user", "aud": server_config.client_id,
"iat": now, "exp": now + 300, "nonce": auth["nonce"],
"email": "user@example.com", "groups": ["readwrite"]
}),
&key,
)
.unwrap();
serde_json::json!({"access_token": "test-access-token", "token_type": "Bearer", "id_token": token})
}
_ => unreachable!(),
}
.to_string();
stream.write_all(format!("HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}", body.len()).as_bytes()).await.unwrap();
}
});
let http_client = ReqwestHttpClient::with_policy(OutboundPolicy::from_allowed_origins(&base).unwrap());
let discovered = discover_provider(&config, &http_client).await.unwrap();
let sys = OidcSys {
configs: HashMap::from([(config.id.clone(), test_sourced_config(config))]),
provider_runtime: ProviderRuntime::new(http_client, HashMap::from([("console".into(), discovered)])),
state_store: OidcStateStore::new(),
};
let auth_url = sys.authorize_url("console", redirect, Some("/buckets".into())).await.unwrap();
let auth_url = Url::parse(&auth_url).unwrap();
assert_eq!(auth_url.as_str().split('?').next(), Some(format!("{base}/authorize").as_str()));
let auth: HashMap<String, String> = auth_url.query_pairs().into_owned().collect();
assert_eq!(auth["response_type"], "code");
assert_eq!(auth["client_id"], "rustfs-oidc-test");
assert!(!auth["nonce"].is_empty());
assert!(!auth["state"].is_empty());
assert_eq!(auth["redirect_uri"], redirect);
assert_eq!(auth["code_challenge_method"], "S256");
assert!(auth["scope"].split_whitespace().any(|scope| scope == "openid"));
let state = auth["state"].clone();
auth_tx.send(auth).unwrap();
let (claims, provider, session, _) = sys
.exchange_code(&state, "test-authorization-code", redirect)
.await
.unwrap_or_else(|err| panic!("hidden={hidden}, explicit={explicit}: {err}"));
assert_eq!(provider, "console");
assert_eq!(claims.sub, "existing-user");
assert_eq!(claims.email, "user@example.com");
assert_eq!(claims.groups, vec!["readwrite"]);
assert_eq!(session.redirect_after.as_deref(), Some("/buckets"));
assert!(matches!(sys.exchange_code(&state, "test-authorization-code", redirect).await,
Err(error) if error == "invalid or expired OIDC state"));
server.await.unwrap();
}
}
}
#[test]
fn test_decode_jwt_payload_invalid() {
assert!(decode_jwt_payload("not-a-jwt").is_empty());
assert!(decode_jwt_payload("").is_empty());
}
#[test]
fn test_core_token_response_accepts_rfc3339_updated_at() {
// Signature verification happens later; this test covers token response deserialization.
let id_token = "eyJhbGciOiJSUzI1NiJ9.eyJpc3MiOiJodHRwczovL2F1dGguZXhhbXBsZS5jb20vb2lkYyIsInN1YiI6InVzZXItMSIsImF1ZCI6InJ1c3RmcyIsImV4cCI6MTc4NDQzMjc2OSwiaWF0IjoxNzgzMjIzMTY5LCJ1cGRhdGVkX2F0IjoiMjAyNi0wNy0wM1QwNDo1MDo1MC44MTFaIn0.c2ln";
let body = serde_json::json!({
"scope": "openid roles profile email",
"token_type": "Bearer",
"access_token": "access-token",
"expires_in": 1209600,
"id_token": id_token,
})
.to_string();
let response: openidconnect::core::CoreTokenResponse =
serde_json::from_str(&body).expect("RFC3339 updated_at should parse in token response");
assert!(response.extra_fields().id_token().is_some());
}
#[test]
fn test_map_claims_to_policies_no_provider() {
let sys = OidcSys::empty().expect("failed to initialize empty OIDC system");
let claims = OidcClaims {
sub: "user123".to_string(),
email: "user@example.com".to_string(),
username: "user".to_string(),
groups: vec!["admin".to_string(), "devs".to_string()],
raw: HashMap::new(),
};
let (policies, groups) = sys.map_claims_to_policies("nonexistent", &claims);
assert!(policies.is_empty());
assert!(groups.is_empty());
}
#[test]
fn test_oidc_claims_default() {
let claims = OidcClaims::default();
assert!(claims.sub.is_empty());
assert!(claims.email.is_empty());
assert!(claims.username.is_empty());
assert!(claims.groups.is_empty());
assert!(claims.raw.is_empty());
}
#[test]
fn test_oidc_claims_serde_roundtrip() {
let claims = OidcClaims {
sub: "user123".to_string(),
email: "user@example.com".to_string(),
username: "testuser".to_string(),
groups: vec!["admin".to_string(), "devs".to_string()],
raw: {
let mut m = HashMap::new();
m.insert("custom".to_string(), serde_json::json!("value"));
m
},
};
let json = serde_json::to_string(&claims).unwrap();
let deserialized: OidcClaims = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.sub, "user123");
assert_eq!(deserialized.email, "user@example.com");
assert_eq!(deserialized.groups.len(), 2);
}
#[test]
fn test_oidc_provider_summary_serde() {
let summary = OidcProviderSummary {
provider_id: "okta".to_string(),
display_name: "Okta SSO".to_string(),
};
let json = serde_json::to_string(&summary).unwrap();
let deserialized: OidcProviderSummary = serde_json::from_str(&json).unwrap();
assert_eq!(deserialized.provider_id, "okta");
assert_eq!(deserialized.display_name, "Okta SSO");
}
#[test]
fn test_oidc_sys_empty() {
let sys = OidcSys::empty().expect("failed to initialize empty OIDC system");
assert!(!sys.has_providers());
assert!(sys.config_snapshot().providers().is_empty());
}
#[tokio::test]
async fn oidc_token_exchange_reports_forbidden_token_endpoint() {
let provider_id = "default";
let token_endpoint = "http://192.168.65.254:8080/realms/rustfs/protocol/openid-connect/token";
let metadata = serde_json::from_value::<ProviderMetadataWithLogout>(serde_json::json!({
"issuer": "https://idp.example.com/realms/rustfs",
"authorization_endpoint": "https://idp.example.com/realms/rustfs/protocol/openid-connect/auth",
"token_endpoint": token_endpoint,
"jwks_uri": "https://idp.example.com/realms/rustfs/protocol/openid-connect/certs",
"response_types_supported": ["code"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["RS256"]
}))
.expect("provider metadata should parse");
let config =
build_mocked_oidc_provider_config(provider_id, "https://idp.example.com/realms/rustfs/.well-known/openid-configuration");
let state_store = OidcStateStore::new();
state_store
.insert(
"test-state".to_string(),
OidcAuthSession {
provider_id: provider_id.to_string(),
pkce_verifier: "test-pkce-verifier".to_string(),
nonce: "test-nonce".to_string(),
redirect_after: None,
},
)
.await;
let sys = OidcSys {
configs: HashMap::from([(provider_id.to_string(), test_sourced_config(config))]),
provider_runtime: ProviderRuntime::new(
ReqwestHttpClient::with_policy(OutboundPolicy::default()),
HashMap::from([(
provider_id.to_string(),
ProviderState {
metadata: DiscoveredProviderMetadata::Console(Box::new(metadata)),
discovered_at: Instant::now(),
},
)]),
),
state_store,
};
let error = match sys
.exchange_code("test-state", "test-code", "https://console.example.com/oauth_callback")
.await
{
Ok(_) => panic!("private token endpoint should require an explicit allowlist origin"),
Err(error) => error,
};
assert!(error.contains("request_error_kind=forbidden_outbound"));
assert!(error.contains(&format!("add http://192.168.65.254:8080 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
}
#[test]
fn test_list_visible_providers_hides_hidden_provider() {
let visible = test_config("dex");
let mut hidden = test_config("kubernetes");
hidden.hide_from_ui = true;
let sys = make_test_sys(vec![visible, hidden]);
let listed = sys.list_visible_providers();
assert_eq!(listed.len(), 1);
assert!(listed.iter().any(|provider| provider.provider_id == "dex"));
assert!(!listed.iter().any(|provider| provider.provider_id == "kubernetes"));
}
#[test]
fn test_hidden_provider_still_resolvable_for_sts() {
let visible = test_config("dex");
let mut hidden = test_config("kubernetes");
hidden.hide_from_ui = true;
let sys = make_test_sys(vec![visible, hidden]);
assert!(sys.get_provider_config("kubernetes").is_some());
assert!(sys.get_provider_config("dex").is_some());
}
#[test]
fn test_config_snapshot_includes_hidden_for_replication() {
let visible = test_config("dex");
let mut hidden = test_config("kubernetes");
hidden.hide_from_ui = true;
let sys = make_test_sys(vec![visible, hidden]);
assert_eq!(sys.config_snapshot().providers().len(), 2);
}
#[test]
fn test_list_providers_includes_hidden_for_compatibility() {
let visible = test_config("dex");
let mut hidden = test_config("kubernetes");
hidden.hide_from_ui = true;
let sys = make_test_sys(vec![visible, hidden]);
assert_eq!(sys.list_providers().len(), 2);
assert_eq!(sys.list_visible_providers().len(), 1);
}
#[test]
fn role_policy_does_not_map_groups_as_policies() {
let mut config = test_config("authentik");
config.role_policy = "consoleAdmin".to_string();
config.claim_name = "policy".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["authentik Admins".to_string(), "users".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("authentik", &claims);
assert_eq!(groups, vec!["authentik Admins", "users"]);
assert_eq!(policies, vec!["consoleAdmin"]);
}
#[test]
fn group_claim_policy_names_cover_only_prefixed_group_values() {
let mut config = test_config("ad");
config.claim_prefix = "oidc-".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec![
"EXAMPLE\\Domain Users".to_string(),
"admins".to_string(),
"admins".to_string(),
],
..Default::default()
};
assert_eq!(
sys.group_claim_policy_names("ad", &claims),
vec!["oidc-EXAMPLE\\Domain Users", "oidc-admins"]
);
assert!(sys.group_claim_policy_names("missing-provider", &claims).is_empty());
}
#[test]
fn group_claim_policy_names_include_merged_roles_claim_values() {
let mut config = test_config("keycloak");
config.roles_claim = "roles".to_string();
let raw = HashMap::from([
("groups".to_string(), serde_json::json!(["readonly"])),
("roles".to_string(), serde_json::json!(["offline_access", "default-roles-corp"])),
]);
let claims = OidcClaims {
groups: extract_canonical_group_values(&raw, &config.groups_claim, &config.roles_claim),
raw,
..Default::default()
};
let sys = make_test_sys(vec![config]);
assert_eq!(
sys.group_claim_policy_names("keycloak", &claims),
vec!["default-roles-corp", "offline_access", "readonly"]
);
}
#[test]
fn group_claim_policy_names_exclude_role_policy_and_dedicated_policy_claim() {
let mut role_config = test_config("role");
role_config.role_policy = "consoleAdmin".to_string();
let mut claim_config = test_config("claim");
claim_config.claim_name = "policy".to_string();
let sys = make_test_sys(vec![role_config, claim_config]);
let claims = OidcClaims {
groups: vec!["readonly".to_string(), "unmapped".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly", "writeonly"]))]),
..Default::default()
};
assert!(sys.group_claim_policy_names("role", &claims).is_empty());
// `readonly` is also requested explicitly through the policy claim, so it must resolve.
assert_eq!(sys.group_claim_policy_names("claim", &claims), vec!["unmapped"]);
}
#[test]
fn test_map_claims_to_policies_with_prefix() {
let mut config = test_config("azure");
config.claim_prefix = "oidc-".to_string();
config.display_name = "Azure AD".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
sub: "user456".to_string(),
email: "user@corp.com".to_string(),
username: "user".to_string(),
groups: vec!["engineers".to_string()],
raw: HashMap::new(),
};
let (policies, groups) = sys.map_claims_to_policies("azure", &claims);
assert_eq!(groups, vec!["engineers"]);
assert!(policies.contains(&"oidc-engineers".to_string()));
assert_eq!(policies.len(), 1);
}
#[test]
fn blank_role_policy_uses_claim_mapping() {
let mut config = test_config("keycloak");
config.role_policy = " ".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["readonly".to_string()],
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["readonly"]);
assert_eq!(policies, vec!["readonly"]);
}
#[test]
fn claim_mapping_keeps_groups_with_distinct_primary_claim() {
let mut config = test_config("keycloak");
config.claim_name = "policy".to_string();
let sys = make_test_sys(vec![config]);
let claims = OidcClaims {
groups: vec!["developers".to_string()],
raw: HashMap::from([("policy".to_string(), serde_json::json!(["readonly"]))]),
..Default::default()
};
let (policies, groups) = sys.map_claims_to_policies("keycloak", &claims);
assert_eq!(groups, vec!["developers"]);
assert_eq!(policies, vec!["developers", "readonly"]);
}
#[test]
fn test_config_snapshot_lists_providers() {
let mut config = test_config("keycloak");
config.display_name = "Keycloak SSO".to_string();
let sys = make_test_sys(vec![config]);
assert!(sys.has_providers());
let snapshot = sys.config_snapshot();
assert_eq!(snapshot.providers().len(), 1);
assert_eq!(snapshot.providers()[0].config.id, "keycloak");
assert_eq!(snapshot.providers()[0].config.display_name, "Keycloak SSO");
assert_eq!(snapshot.providers()[0].source, OidcProviderConfigSource::Persisted);
}
#[test]
fn test_get_provider_config() {
let mut config = test_config("test");
config.client_id = "my-client".to_string();
config.client_secret = Some("secret".to_string());
let sys = make_test_sys(vec![config]);
assert!(sys.get_provider_config("test").is_some());
assert_eq!(sys.get_provider_config("test").unwrap().client_id, "my-client");
assert!(sys.get_provider_config("nonexistent").is_none());
}
#[test]
fn verified_identity_preserves_provider_and_claim_values() {
let raw = HashMap::from([
("iss".to_string(), serde_json::json!("https://corp.example.test")),
("department".to_string(), serde_json::json!("engineering")),
("roles".to_string(), serde_json::json!(["reader", "admin"])),
]);
let identity = verified_identity(
FederatedProviderRef::new("corp".to_string()),
OidcClaims {
sub: " subject-123 ".to_string(),
email: " user@example.test ".to_string(),
username: " user ".to_string(),
groups: vec![
"source-ops".to_string(),
"source-developers".to_string(),
"source-ops".to_string(),
],
raw: raw.clone(),
},
);
assert_eq!(identity.provider().as_str(), "corp");
assert_eq!(identity.subject(), " subject-123 ");
assert_eq!(identity.email(), " user@example.test ");
assert_eq!(identity.username(), " user ");
assert_eq!(identity.source_groups(), ["source-ops", "source-developers", "source-ops"]);
assert_eq!(identity.attributes(), &raw);
}
#[test]
fn adapter_preserves_provider_views_and_redirect_policy() {
let mut visible = test_config("default");
visible.display_name = "Company SSO".to_string();
visible.redirect_uri = Some("https://console.example.test/callback".to_string());
visible.redirect_uri_dynamic = false;
let mut hidden = test_config("workload");
hidden.display_name = "Workload Identity".to_string();
hidden.hide_from_ui = true;
let adapter = StandardOidcAdapter::new(Arc::new(make_test_sys(vec![visible, hidden])));
let mut providers = adapter.list_providers();
providers.sort_by(|left, right| left.provider_id.cmp(&right.provider_id));
assert_eq!(
providers,
vec![
FederatedProviderView {
provider_id: "default".to_string(),
display_name: "Company SSO".to_string(),
},
FederatedProviderView {
provider_id: "workload".to_string(),
display_name: "Workload Identity".to_string(),
},
]
);
assert_eq!(
adapter.list_visible_providers(),
vec![FederatedProviderView {
provider_id: "default".to_string(),
display_name: "Company SSO".to_string(),
}]
);
assert_eq!(
adapter.redirect_policy("default"),
Some(FederatedRedirectPolicy {
redirect_uri: Some("https://console.example.test/callback".to_string()),
allow_request_origin: false,
})
);
assert_eq!(adapter.redirect_policy("missing"), None);
}
crates/iam/src/oidc_state.rs → crates/iam/src/oidc/state.rs
+18 -16
View File
@@ -25,32 +25,32 @@ const OIDC_STATE_CAPACITY_LOG_INTERVAL_SECS: u64 = 60;
/// Stores the PKCE verifier and nonce for an in-flight OIDC authorization flow.
#[derive(Debug, Clone)]
pub struct OidcAuthSession {
pub provider_id: String,
pub pkce_verifier: String,
pub nonce: String,
pub redirect_after: Option<String>,
pub(super) struct OidcAuthSession {
pub(super) provider_id: String,
pub(super) pkce_verifier: String,
pub(super) nonce: String,
pub(super) redirect_after: Option<String>,
}
/// Stores an ID token behind a one-time opaque handle so the console can trigger
/// RP-initiated logout without persisting the raw token in browser storage.
#[derive(Debug, Clone)]
pub struct OidcLogoutSession {
pub provider_id: String,
pub id_token: String,
pub(super) struct OidcLogoutSession {
pub(super) provider_id: String,
pub(super) id_token: String,
}
/// TTL cache for OIDC auth state (PKCE verifiers + nonces) during the authorization flow.
/// Entries expire after 5 minutes and are single-use (removed on retrieval).
#[derive(Clone)]
pub struct OidcStateStore {
pub(super) struct OidcStateStore {
cache: Cache<String, OidcAuthSession>,
logout_cache: Cache<String, OidcLogoutSession>,
last_capacity_log_at: Arc<AtomicU64>,
}
impl OidcStateStore {
pub fn new() -> Self {
pub(super) fn new() -> Self {
let cache = Cache::builder()
.max_capacity(OIDC_STATE_CAPACITY)
.time_to_live(Duration::from_secs(300)) // 5 minute TTL
@@ -67,7 +67,7 @@ impl OidcStateStore {
}
/// Store a new auth session keyed by the OAuth2 `state` parameter.
pub async fn insert(&self, state: String, session: OidcAuthSession) {
pub(super) async fn insert(&self, state: String, session: OidcAuthSession) {
self.cache.insert(state, session).await;
let size = self.cache.entry_count();
@@ -104,27 +104,29 @@ impl OidcStateStore {
/// Retrieve and remove an auth session (single-use). Returns None if expired or not found.
/// Uses `remove` which returns the value and removes it in a single operation.
pub async fn take(&self, state: &str) -> Option<OidcAuthSession> {
pub(super) async fn take(&self, state: &str) -> Option<OidcAuthSession> {
self.cache.remove(state).await
}
/// Check if a state key exists (without consuming it).
pub async fn contains(&self, state: &str) -> bool {
#[cfg(test)]
pub(super) async fn contains(&self, state: &str) -> bool {
self.cache.get(state).await.is_some()
}
/// Store a new one-time logout session keyed by an opaque logout token.
pub async fn insert_logout(&self, token: String, session: OidcLogoutSession) {
pub(super) async fn insert_logout(&self, token: String, session: OidcLogoutSession) {
self.logout_cache.insert(token, session).await;
}
/// Retrieve and remove a logout session (single-use). Returns None if expired or not found.
pub async fn take_logout(&self, token: &str) -> Option<OidcLogoutSession> {
pub(super) async fn take_logout(&self, token: &str) -> Option<OidcLogoutSession> {
self.logout_cache.remove(token).await
}
/// Check if a logout token exists (without consuming it).
pub async fn contains_logout(&self, token: &str) -> bool {
#[cfg(test)]
pub(super) async fn contains_logout(&self, token: &str) -> bool {
self.logout_cache.get(token).await.is_some()
}
}
crates/iam/src/oidc/test_support.rs
+526
View File
@@ -0,0 +1,526 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::{
config::{OidcProviderConfig, OidcProviderConfigSource, SourcedOidcProviderConfig},
provider::ProviderRuntime,
runtime::OidcSys,
state::OidcStateStore,
transport::ReqwestHttpClient,
};
use std::collections::HashMap;
#[cfg(any(test, feature = "test-util"))]
#[doc(hidden)]
pub fn make_test_sys(configs: Vec<OidcProviderConfig>) -> OidcSys {
let configs = configs
.into_iter()
.map(test_sourced_config)
.map(|provider| (provider.config.id.clone(), provider))
.collect();
OidcSys {
configs,
provider_runtime: ProviderRuntime::new(
ReqwestHttpClient::new().expect("failed to initialize OIDC HTTP clients"),
HashMap::new(),
),
state_store: OidcStateStore::new(),
}
}
#[cfg(any(test, feature = "test-util"))]
pub(super) fn test_sourced_config(config: OidcProviderConfig) -> SourcedOidcProviderConfig {
SourcedOidcProviderConfig {
config,
source: OidcProviderConfigSource::Persisted,
}
}
#[cfg(any(test, feature = "test-util"))]
#[doc(hidden)]
pub fn test_config(id: &str) -> OidcProviderConfig {
OidcProviderConfig {
id: id.to_string(),
enabled: true,
config_url: format!("https://example.com/{id}/.well-known/openid-configuration"),
issuer: None,
client_id: "client-id".to_string(),
client_secret: None,
scopes: vec!["openid".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: "groups".to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
display_name: id.to_string(),
groups_claim: "groups".to_string(),
roles_claim: String::new(),
email_claim: "email".to_string(),
username_claim: "preferred_username".to_string(),
hide_from_ui: false,
}
}
#[cfg(test)]
use super::{config::OidcProviderValidationResult, provider::discover_provider, transport::OidcHttpError};
#[cfg(test)]
use jsonwebtoken::{Algorithm, EncodingKey};
#[cfg(test)]
use openidconnect::AsyncHttpClient;
#[cfg(test)]
use rustfs_utils::egress::{OutboundDnsPolicyRejection, OutboundPolicy};
#[cfg(test)]
use std::sync::Arc;
#[cfg(test)]
use std::sync::atomic::{AtomicUsize, Ordering};
#[cfg(test)]
use url::Url;
#[cfg(test)]
#[derive(Clone)]
pub(super) struct RejectingDnsResolver {
pub(super) allow_origin_can_recover: bool,
pub(super) calls: Option<Arc<AtomicUsize>>,
}
#[cfg(test)]
impl reqwest::dns::Resolve for RejectingDnsResolver {
fn resolve(&self, name: reqwest::dns::Name) -> reqwest::dns::Resolving {
if let Some(calls) = &self.calls {
calls.fetch_add(1, Ordering::Relaxed);
}
let host = name.as_str().to_string();
let rejection = OutboundDnsPolicyRejection::new(host, self.allow_origin_can_recover);
Box::pin(async move { Err(std::io::Error::new(std::io::ErrorKind::PermissionDenied, rejection).into()) })
}
}
#[cfg(test)]
pub(super) fn build_mocked_oidc_provider_config(id: &str, config_url: &str) -> OidcProviderConfig {
OidcProviderConfig {
id: id.to_string(),
enabled: true,
config_url: config_url.to_string(),
issuer: None,
client_id: "rustfs-oidc-test".to_string(),
client_secret: None,
scopes: vec!["openid".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: false,
claim_name: "sub".to_string(),
claim_prefix: "oidc".to_string(),
role_policy: String::new(),
display_name: "mock-oidc".to_string(),
groups_claim: "groups".to_string(),
roles_claim: String::new(),
email_claim: "email".to_string(),
username_claim: "username".to_string(),
hide_from_ui: false,
}
}
#[cfg(test)]
pub(super) fn read_mock_oidc_request(stream: &mut impl std::io::Read) -> String {
let mut request_bytes = Vec::new();
let mut buffer = [0u8; 4096];
loop {
match stream.read(&mut buffer) {
Ok(0) => break,
Ok(n) => request_bytes.extend_from_slice(&buffer[..n]),
Err(e) if matches!(e.kind(), std::io::ErrorKind::WouldBlock | std::io::ErrorKind::TimedOut) => break,
Err(_) => break,
}
if request_bytes.windows(4).any(|w| w == b"\r\n\r\n") {
break;
}
if request_bytes.len() >= 8192 {
break;
}
}
String::from_utf8_lossy(&request_bytes).into_owned()
}
#[cfg(test)]
pub(super) fn read_mock_oidc_request_path(stream: &mut impl std::io::Read) -> String {
read_mock_oidc_request(stream)
.lines()
.next()
.unwrap_or("")
.split_whitespace()
.nth(1)
.unwrap_or("")
.to_string()
}
#[cfg(test)]
pub(super) fn mock_oidc_response(path: &str, discovery_body: &str, expected_jwks_path: &str, jwks_body: &str) -> String {
let (status, body) = if path.ends_with("/.well-known/openid-configuration") {
(200, discovery_body)
} else if path == expected_jwks_path {
(200, jwks_body)
} else {
(404, r#"{"error":"not found"}"#)
};
format!(
"HTTP/1.1 {status} {}\r\nContent-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
if status == 200 { "OK" } else { "Not Found" },
body.len()
)
}
#[cfg(test)]
pub(super) fn start_mock_oidc_discovery_server_with_jwks<F, J>(
build_discovery_issuer: F,
max_requests: usize,
signing_alg: &'static str,
workload: bool,
jwks_response: J,
) -> Option<(String, std::thread::JoinHandle<()>)>
where
F: Fn(&str) -> (String, String, String) + Send + 'static,
J: Fn(usize) -> String + Send + 'static,
{
use std::io::Write;
use std::net::{Shutdown, TcpListener};
use std::sync::mpsc;
use std::time::{Duration, Instant};
// After the last completed response, exit if no new connection arrives within this window.
// Keep the mock server alive long enough for slower CI/macOS test environments to finish
// discovery + JWKS requests without racing the shutdown timer.
const IDLE_SHUTDOWN: Duration = Duration::from_secs(1);
const ABSOLUTE_CAP: Duration = Duration::from_secs(5);
let listener = match TcpListener::bind("127.0.0.1:0") {
Ok(listener) => listener,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
Err(err) => panic!("test listener should bind: {err}"),
};
let base = format!("http://{}", listener.local_addr().expect("listener local address should be available"));
let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base);
let mut discovery_document = serde_json::json!({
"issuer": discovery_issuer,
"authorization_endpoint": format!("{base}/authorize"),
"token_endpoint": format!("{base}/token"),
"jwks_uri": discovery_jwks_uri,
"response_types_supported": ["code"],
"response_modes_supported": ["query"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": [signing_alg],
});
if workload {
let fields = discovery_document.as_object_mut().expect("mock metadata is an object");
fields.remove("authorization_endpoint");
fields.remove("token_endpoint");
fields.insert("response_types_supported".into(), serde_json::json!(["id_token"]));
}
let discovery_body = discovery_document.to_string();
let (ready_tx, ready_rx) = mpsc::channel();
let handle = std::thread::spawn(move || {
listener
.set_nonblocking(true)
.expect("failed to set discovery mock listener non-blocking");
let _ = ready_tx.send(());
let mut seen = 0usize;
let mut jwks_fetches = 0usize;
let start = Instant::now();
let mut last_completed = Instant::now();
loop {
if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN {
break;
}
if start.elapsed() >= ABSOLUTE_CAP {
break;
}
let mut stream = match listener.accept() {
Ok((stream, _)) => stream,
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(5));
continue;
}
Err(_) => break,
};
stream
.set_nonblocking(false)
.expect("failed to set discovery mock stream blocking");
seen += 1;
stream
.set_nonblocking(false)
.expect("failed to set discovery mock stream blocking");
stream
.set_read_timeout(Some(Duration::from_secs(1)))
.expect("failed to set discovery mock read timeout");
let request = read_mock_oidc_request(&mut stream);
let path = request
.lines()
.next()
.and_then(|line| line.split_whitespace().nth(1))
.unwrap_or("");
let jwks_body = jwks_response(jwks_fetches);
if path == expected_jwks_path {
jwks_fetches += 1;
}
let mut response = mock_oidc_response(path, &discovery_body, &expected_jwks_path, &jwks_body);
if path.contains("/.well-known/openid-configuration") {
assert!(
request
.lines()
.filter_map(|line| line.split_once(':'))
.any(|(name, value)| { name.eq_ignore_ascii_case("accept") && value.trim() == "application/json" }),
"discovery Accept must remain unchanged"
);
}
if path == expected_jwks_path {
let expected_type = if workload {
"application/jwk-set+json"
} else {
"application/json"
};
let accepts_type = request.lines().filter_map(|line| line.split_once(':')).any(|(name, value)| {
name.eq_ignore_ascii_case("accept") && value.split(',').any(|item| item.trim() == expected_type)
});
if !accepts_type {
response = "HTTP/1.1 406 Not Acceptable\r\nContent-Length: 0\r\nConnection: close\r\n\r\n".into();
} else if workload {
response = response.replace("Content-Type: application/json", "Content-Type: application/jwk-set+json");
}
}
let _ = stream.write_all(response.as_bytes());
let _ = stream.flush();
let _ = stream.shutdown(Shutdown::Both);
last_completed = Instant::now();
if seen >= max_requests {
break;
}
}
});
ready_rx
.recv_timeout(Duration::from_millis(100))
.expect("mock OIDC discovery server should become ready");
Some((base, handle))
}
#[cfg(test)]
pub(super) fn start_mock_oidc_discovery_server<F>(
build_discovery_issuer: F,
max_requests: usize,
) -> Option<(String, std::thread::JoinHandle<()>)>
where
F: Fn(&str) -> (String, String, String) + Send + 'static,
{
start_mock_oidc_discovery_server_with_jwks(build_discovery_issuer, max_requests, "RS256", false, |_| {
r#"{"keys":[]}"#.to_string()
})
}
#[cfg(test)]
pub(super) fn oidc_es256_key_and_jwk(kid: &str) -> (EncodingKey, serde_json::Value) {
let certified = rcgen::generate_simple_self_signed(vec![format!("{kid}.invalid")]).expect("OIDC signing key should generate");
let encoding_key =
EncodingKey::from_ec_pem(certified.signing_key.serialize_pem().as_bytes()).expect("OIDC signing key should encode");
let mut jwk = jsonwebtoken::jwk::Jwk::from_encoding_key(&encoding_key, Algorithm::ES256)
.expect("OIDC public JWK should derive from signing key");
jwk.common.key_id = Some(kid.to_string());
jwk.common.public_key_use = Some(jsonwebtoken::jwk::PublicKeyUse::Signature);
(encoding_key, serde_json::to_value(jwk).expect("OIDC JWK should serialize"))
}
#[cfg(test)]
pub(super) fn start_mock_oidc_tls_discovery_server<F>(
build_discovery_issuer: F,
max_requests: usize,
) -> Option<(String, String, std::thread::JoinHandle<()>)>
where
F: Fn(&str) -> (String, String, String) + Send + 'static,
{
use std::io::Write;
use std::net::{Shutdown, TcpListener};
use std::sync::mpsc;
use std::time::{Duration, Instant};
const IDLE_SHUTDOWN: Duration = Duration::from_secs(1);
const ABSOLUTE_CAP: Duration = Duration::from_secs(5);
let _ = rustls::crypto::aws_lc_rs::default_provider().install_default();
let certified =
rcgen::generate_simple_self_signed(vec!["127.0.0.1".to_string()]).expect("generate OIDC TLS test certificate");
let cert_pem = certified.cert.pem();
let server_config = rustls::ServerConfig::builder()
.with_no_client_auth()
.with_single_cert(
vec![certified.cert.der().clone()],
rustls_pki_types::PrivateKeyDer::try_from(certified.signing_key.serialize_der())
.expect("convert OIDC TLS test private key"),
)
.expect("build OIDC TLS mock server config");
let listener = match TcpListener::bind("127.0.0.1:0") {
Ok(listener) => listener,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
Err(err) => panic!("test TLS listener should bind: {err}"),
};
let base = format!("https://{}", listener.local_addr().expect("listener local address should be available"));
let (discovery_issuer, discovery_jwks_uri, expected_jwks_path) = build_discovery_issuer(&base);
let discovery_body = serde_json::json!({
"issuer": discovery_issuer,
"authorization_endpoint": format!("{base}/authorize"),
"token_endpoint": format!("{base}/token"),
"jwks_uri": discovery_jwks_uri,
"response_types_supported": ["code"],
"response_modes_supported": ["query"],
"subject_types_supported": ["public"],
"id_token_signing_alg_values_supported": ["RS256"],
})
.to_string();
let jwks_body = r#"{"keys":[]}"#;
let (ready_tx, ready_rx) = mpsc::channel();
let handle = std::thread::spawn(move || {
let server_config = Arc::new(server_config);
listener
.set_nonblocking(true)
.expect("failed to set TLS discovery mock listener non-blocking");
let _ = ready_tx.send(());
let mut seen = 0usize;
let start = Instant::now();
let mut last_completed = Instant::now();
loop {
if seen > 0 && last_completed.elapsed() >= IDLE_SHUTDOWN {
break;
}
if start.elapsed() >= ABSOLUTE_CAP {
break;
}
let tcp_stream = match listener.accept() {
Ok((stream, _)) => stream,
Err(e) if e.kind() == std::io::ErrorKind::WouldBlock => {
std::thread::sleep(Duration::from_millis(5));
continue;
}
Err(_) => break,
};
tcp_stream
.set_nonblocking(false)
.expect("failed to set TLS discovery mock stream blocking");
tcp_stream
.set_read_timeout(Some(Duration::from_secs(1)))
.expect("failed to set TLS discovery mock read timeout");
seen += 1;
let connection = match rustls::ServerConnection::new(server_config.clone()) {
Ok(connection) => connection,
Err(_) => break,
};
let mut stream = rustls::StreamOwned::new(connection, tcp_stream);
let path = read_mock_oidc_request_path(&mut stream);
let response = mock_oidc_response(&path, &discovery_body, &expected_jwks_path, jwks_body);
let _ = stream.write_all(response.as_bytes());
let _ = stream.flush();
let _ = stream.sock.shutdown(Shutdown::Both);
last_completed = Instant::now();
if seen >= max_requests {
break;
}
}
});
ready_rx
.recv_timeout(Duration::from_millis(100))
.expect("mock TLS OIDC discovery server should become ready");
Some((base, cert_pem, handle))
}
#[cfg(test)]
pub(super) fn discovery_error_contains_all_variants(err: &str, base: &str) -> bool {
err.contains(base) && err.contains(&format!("{base}/")) && err.contains("discovery failed for all issuer variants")
}
#[cfg(test)]
pub(super) async fn validate_mocked_oidc_provider_config(
config: &OidcProviderConfig,
) -> Result<OidcProviderValidationResult, String> {
// The mock discovery/JWKS/token endpoints share the loopback origin of `config_url`.
// Explicitly allow that origin so the egress policy does not reject the loopback mock.
let origin = Url::parse(&config.config_url)
.map_err(|_| "invalid mock config_url".to_string())?
.origin()
.ascii_serialization();
let policy = OutboundPolicy::from_allowed_origins(&origin).map_err(|err| err.to_string())?;
let http_client = ReqwestHttpClient::with_policy(policy);
let state = discover_provider(config, &http_client).await?;
Ok(OidcProviderValidationResult {
issuer: state.metadata.issuer().to_string(),
authorization_endpoint: state.metadata.authorization_endpoint(),
token_endpoint: state.metadata.token_endpoint().map(ToString::to_string),
})
}
#[cfg(test)]
pub(super) fn start_unbounded_body_server(body_len: usize) -> Option<(String, std::thread::JoinHandle<()>)> {
use std::io::{Read, Write};
use std::net::TcpListener;
use std::sync::mpsc;
use std::time::Duration;
let listener = match TcpListener::bind("127.0.0.1:0") {
Ok(listener) => listener,
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
Err(err) => panic!("test listener should bind: {err}"),
};
let base = format!("http://{}", listener.local_addr().expect("listener local address should be available"));
let (ready_tx, ready_rx) = mpsc::channel();
let handle = std::thread::spawn(move || {
let _ = ready_tx.send(());
let Ok((mut stream, _)) = listener.accept() else {
return;
};
let _ = stream.set_read_timeout(Some(Duration::from_secs(1)));
let mut buffer = [0u8; 4096];
let _ = stream.read(&mut buffer);
let _ = stream.write_all(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\nConnection: close\r\n\r\n");
let chunk = vec![b'a'; 64 * 1024];
let mut written = 0usize;
while written < body_len {
let take = chunk.len().min(body_len - written);
if stream.write_all(&chunk[..take]).is_err() {
break;
}
written += take;
}
let _ = stream.flush();
});
ready_rx
.recv_timeout(Duration::from_millis(100))
.expect("mock body server should become ready");
Some((base, handle))
}
#[cfg(test)]
pub(super) async fn fetch_oidc_mock_body(base: &str) -> Result<Vec<u8>, OidcHttpError> {
let policy = OutboundPolicy::from_allowed_origins(base).expect("origin should parse");
let client = ReqwestHttpClient::with_policy(policy);
let request = http::Request::builder()
.method(http::Method::GET)
.uri(base)
.body(Vec::new())
.expect("request should build");
client.call(request).await.map(http::Response::into_body)
}
crates/iam/src/oidc/transport.rs
+660
View File
@@ -0,0 +1,660 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::{LOG_COMPONENT_IAM, LOG_SUBSYSTEM_OIDC};
use openidconnect::AsyncHttpClient;
use reqwest::{Certificate, Client};
use rustfs_config::MAX_OIDC_RESPONSE_SIZE;
use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy, find_outbound_dns_policy_rejection};
use std::collections::VecDeque;
use std::future::Future;
use std::net::IpAddr;
use std::pin::Pin;
use std::sync::{Arc, LazyLock, Mutex, MutexGuard, RwLock};
use std::time::{Duration as StdDuration, Instant};
use tracing::{debug, error, warn};
use url::Url;
pub(super) const EVENT_OIDC_HTTP: &str = "oidc_http";
const OIDC_HTTP_REQUEST_TIMEOUT: StdDuration = StdDuration::from_secs(10);
const OIDC_HTTP_CONNECT_TIMEOUT: StdDuration = StdDuration::from_secs(3);
const OIDC_PLUGIN_AUTHN_WINDOW: StdDuration = StdDuration::from_secs(60);
#[derive(Debug, Clone, Copy, Default)]
pub struct OidcPluginAuthnMetricsSnapshot {
pub failed_requests_minute: u64,
pub last_fail_seconds: u64,
pub last_succ_seconds: u64,
pub succ_avg_rtt_ms_minute: u64,
pub succ_max_rtt_ms_minute: u64,
pub total_requests_minute: u64,
}
#[derive(Debug, Clone)]
struct OidcPluginAuthnSample {
observed_at: Instant,
succeeded: bool,
rtt_ms: u64,
}
#[derive(Debug, Default)]
struct OidcPluginAuthnMetrics {
samples: Mutex<VecDeque<OidcPluginAuthnSample>>,
last_fail_at: Mutex<Option<Instant>>,
last_succ_at: Mutex<Option<Instant>>,
}
fn lock_oidc_plugin_authn_metrics<'a, T>(mutex: &'a Mutex<T>, metric: &'static str) -> MutexGuard<'a, T> {
match mutex.lock() {
Ok(guard) => guard,
Err(err) => {
warn!(metric, "Recovering poisoned OIDC authn metrics lock");
err.into_inner()
}
}
}
fn seconds_since(now: Instant, observed_at: Option<Instant>) -> u64 {
observed_at
.map(|instant| now.duration_since(instant).as_secs())
.unwrap_or_default()
}
impl OidcPluginAuthnMetrics {
fn record(&self, rtt_ms: u64, succeeded: bool) {
let now = Instant::now();
let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples");
samples.push_back(OidcPluginAuthnSample {
observed_at: now,
succeeded,
rtt_ms,
});
while samples
.front()
.is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW)
{
samples.pop_front();
}
drop(samples);
if succeeded {
*lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at") = Some(now);
} else {
*lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at") = Some(now);
}
}
fn snapshot(&self) -> OidcPluginAuthnMetricsSnapshot {
let now = Instant::now();
let (total_requests_minute, failed_requests_minute, succ_avg_rtt_ms_minute, succ_max_rtt_ms_minute) = {
let mut samples = lock_oidc_plugin_authn_metrics(&self.samples, "samples");
while samples
.front()
.is_some_and(|sample| now.duration_since(sample.observed_at) > OIDC_PLUGIN_AUTHN_WINDOW)
{
samples.pop_front();
}
let mut failed_requests_minute = 0u64;
let mut successful_requests = 0u64;
let mut successful_rtt_sum = 0u64;
let mut succ_max_rtt_ms_minute = 0u64;
for sample in samples.iter() {
if sample.succeeded {
successful_requests += 1;
successful_rtt_sum += sample.rtt_ms;
succ_max_rtt_ms_minute = succ_max_rtt_ms_minute.max(sample.rtt_ms);
} else {
failed_requests_minute += 1;
}
}
let succ_avg_rtt_ms_minute = successful_rtt_sum.checked_div(successful_requests).unwrap_or_default();
(
samples.len() as u64,
failed_requests_minute,
succ_avg_rtt_ms_minute,
succ_max_rtt_ms_minute,
)
};
let last_fail_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_fail_at, "last_fail_at"));
let last_succ_seconds = seconds_since(now, *lock_oidc_plugin_authn_metrics(&self.last_succ_at, "last_succ_at"));
OidcPluginAuthnMetricsSnapshot {
failed_requests_minute,
last_fail_seconds,
last_succ_seconds,
succ_avg_rtt_ms_minute,
succ_max_rtt_ms_minute,
total_requests_minute,
}
}
}
static OIDC_PLUGIN_AUTHN_METRICS: LazyLock<OidcPluginAuthnMetrics> = LazyLock::new(OidcPluginAuthnMetrics::default);
pub fn oidc_plugin_authn_metrics_snapshot() -> OidcPluginAuthnMetricsSnapshot {
OIDC_PLUGIN_AUTHN_METRICS.snapshot()
}
/// Header names whose values may carry OIDC secrets (client credentials, cookies,
/// bearer tokens). Their values are never emitted to logs, only their byte length.
const SENSITIVE_HEADER_NAMES: [&str; 4] = ["authorization", "proxy-authorization", "cookie", "set-cookie"];
pub(super) fn is_sensitive_header(name: &str) -> bool {
SENSITIVE_HEADER_NAMES
.iter()
.any(|candidate| name.eq_ignore_ascii_case(candidate))
}
pub(super) fn format_http_headers(headers: &http::HeaderMap) -> String {
headers
.iter()
.map(|(name, value)| {
if is_sensitive_header(name.as_str()) {
format!("{}=<redacted len={}>", name.as_str(), value.as_bytes().len())
} else {
let value = value.to_str().unwrap_or("<non-utf8>");
format!("{}={}", name.as_str(), value)
}
})
.collect::<Vec<_>>()
.join("; ")
}
#[derive(Debug, Default)]
pub(super) struct TokenResponseBodyShape {
pub(super) json_object: bool,
pub(super) json_keys: String,
pub(super) has_access_token: bool,
pub(super) has_id_token: bool,
pub(super) has_token_type: bool,
pub(super) has_expires_in: bool,
pub(super) has_error: bool,
pub(super) has_error_description: bool,
pub(super) looks_like_html: bool,
}
pub(super) fn inspect_token_response_body(body: &[u8]) -> TokenResponseBodyShape {
let mut shape = TokenResponseBodyShape {
looks_like_html: body
.iter()
.copied()
.find(|byte| !byte.is_ascii_whitespace())
.is_some_and(|byte| byte == b'<'),
..Default::default()
};
let Ok(value) = serde_json::from_slice::<serde_json::Value>(body) else {
return shape;
};
let Some(object) = value.as_object() else {
return shape;
};
shape.json_object = true;
shape.has_access_token = object.contains_key("access_token");
shape.has_id_token = object.contains_key("id_token");
shape.has_token_type = object.contains_key("token_type");
shape.has_expires_in = object.contains_key("expires_in");
shape.has_error = object.contains_key("error");
shape.has_error_description = object.contains_key("error_description");
let mut keys: Vec<&str> = object.keys().map(String::as_str).collect();
keys.sort_unstable();
keys.truncate(16);
shape.json_keys = keys.join(",");
shape
}
pub(super) fn oidc_http_error_diagnostics(error: &OidcHttpError) -> (&'static str, String) {
match error {
OidcHttpError::Reqwest(err) if err.is_timeout() => ("timeout", String::new()),
OidcHttpError::Reqwest(err) if err.is_connect() => ("connect", String::new()),
OidcHttpError::Reqwest(err) if err.status().is_some() => {
("http_status", err.status().map(|status| status.as_u16().to_string()).unwrap_or_default())
}
OidcHttpError::Reqwest(_) => ("request", String::new()),
OidcHttpError::Http(_) => ("http_build", String::new()),
OidcHttpError::ExtraRootCa(_) => ("extra_root_ca", String::new()),
OidcHttpError::ForbiddenOutbound(_) => ("forbidden_outbound", String::new()),
OidcHttpError::ResponseTooLarge(limit) => ("response_too_large", limit.to_string()),
}
}
// ---- HTTP Client Adapter ----
/// Error type for the OIDC HTTP client adapter.
#[derive(Debug)]
pub(super) enum OidcHttpError {
Reqwest(reqwest::Error),
Http(http::Error),
ExtraRootCa(String),
/// The outbound destination was rejected by the shared egress policy before any
/// connection was attempted (invalid URL, loopback/link-local/metadata/private IP,
/// or a malformed allow-origins configuration).
ForbiddenOutbound(String),
/// The provider response body exceeded [`MAX_OIDC_RESPONSE_SIZE`] and was abandoned
/// instead of being buffered in full.
ResponseTooLarge(usize),
}
impl std::fmt::Display for OidcHttpError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::Reqwest(e) => write!(f, "{e}"),
Self::Http(e) => write!(f, "{e}"),
Self::ExtraRootCa(reason) => write!(f, "failed to load OIDC extra root CA bundle: {reason}"),
Self::ForbiddenOutbound(reason) => write!(f, "outbound request rejected: {reason}"),
Self::ResponseTooLarge(limit) => write!(f, "oidc response body exceeds {limit} bytes"),
}
}
}
impl std::error::Error for OidcHttpError {
fn source(&self) -> Option<&(dyn std::error::Error + 'static)> {
match self {
Self::Reqwest(e) => Some(e),
Self::Http(e) => Some(e),
Self::ExtraRootCa(_) | Self::ForbiddenOutbound(_) | Self::ResponseTooLarge(_) => None,
}
}
}
#[derive(Clone, Debug, Default)]
pub struct OidcExtraRootCaMaterial {
pub generation: u64,
pub root_ca_pem: Option<Vec<u8>>,
}
type OidcExtraRootCaFuture = Pin<Box<dyn Future<Output = Result<OidcExtraRootCaMaterial, String>> + Send>>;
type OidcExtraRootCaLoader = dyn Fn() -> OidcExtraRootCaFuture + Send + Sync;
#[derive(Clone)]
pub struct OidcExtraRootCaProvider {
loader: Arc<OidcExtraRootCaLoader>,
}
impl OidcExtraRootCaProvider {
pub fn new<F, Fut>(loader: F) -> Self
where
F: Fn() -> Fut + Send + Sync + 'static,
Fut: Future<Output = Result<OidcExtraRootCaMaterial, String>> + Send + 'static,
{
Self {
loader: Arc::new(move || Box::pin(loader())),
}
}
async fn load(&self) -> Result<OidcExtraRootCaMaterial, String> {
(self.loader)().await
}
}
#[derive(Clone, Default)]
struct CachedOidcExtraRootCerts {
generation: u64,
initialized: bool,
certs: Vec<Certificate>,
}
/// HTTP client adapter bridging reqwest 0.13 to the `openidconnect` `AsyncHttpClient` trait.
///
/// A fresh client is built for every request so the destination is re-validated and the
/// resolved IP is re-classified at connection time. This closes the SSRF / DNS-rebinding
/// gap where a one-shot URL string check is bypassed by a hostname that resolves to an
/// internal address only at connection time, and it also covers endpoints discovered from
/// the provider metadata (JWKS, token) rather than only the operator-configured `config_url`.
pub(super) struct ReqwestHttpClient {
/// `None` in production: the process-cached outbound policy from the environment is used.
/// `Some(..)` only in tests, to explicitly allow a loopback mock endpoint.
policy_override: Option<OutboundPolicy>,
extra_root_certs: Arc<RwLock<CachedOidcExtraRootCerts>>,
extra_root_ca_provider: Option<OidcExtraRootCaProvider>,
#[cfg(test)]
dns_resolver_override: Option<Arc<dyn reqwest::dns::Resolve>>,
}
pub(super) fn parse_oidc_extra_root_certs(source: &str, pem: &[u8]) -> Result<Vec<Certificate>, String> {
if pem.iter().all(|byte| byte.is_ascii_whitespace()) {
return Ok(Vec::new());
}
Certificate::from_pem_bundle(pem).map_err(|err| format!("failed to parse OIDC extra root CA bundle from {source}: {err}"))
}
pub(super) fn oidc_extra_root_certs(root_ca_pem: Option<&[u8]>) -> Result<Vec<Certificate>, String> {
match root_ca_pem {
Some(pem) => parse_oidc_extra_root_certs("RustFS outbound TLS material", pem),
None => Ok(Vec::new()),
}
}
/// Build a reqwest client pinned to the shared outbound egress policy for a single request.
///
/// [`OutboundPolicy::resolver_for`] validates the URL shape and rejects loopback,
/// link-local, metadata, multicast and unauthorized private addresses up front, and the
/// returned `OutboundDnsResolver` re-resolves and re-classifies the host on every new
/// connection so DNS rebinding fails closed. Redirects are not followed: a redirect target
/// would otherwise skip URL-shape re-validation. The timeouts bound how long a slow or
/// stalled provider can pin the calling task.
pub(super) fn build_oidc_http_client(
uri: &str,
policy_override: Option<&OutboundPolicy>,
extra_root_certs: &[Certificate],
#[cfg(test)] dns_resolver_override: Option<Arc<dyn reqwest::dns::Resolve>>,
) -> Result<(Client, Url), OidcHttpError> {
let url = Url::parse(uri).map_err(|_| OidcHttpError::ForbiddenOutbound("invalid outbound OIDC URL".to_string()))?;
let resolver = match policy_override {
Some(policy) => policy.resolver_for(&url),
None => OutboundPolicy::from_env_cached()
.map_err(|err| OidcHttpError::ForbiddenOutbound(err.to_string()))?
.resolver_for(&url),
}
.map_err(|err| {
let base = err.to_string();
let origin = url.origin().ascii_serialization();
let can_allow_origin =
OutboundPolicy::from_allowed_origins(&origin).is_ok_and(|allowlisted| allowlisted.validate_url(&url).is_ok());
oidc_forbidden_outbound_error(&url, base, can_allow_origin)
})?;
let bypass_proxy = should_bypass_proxy_for_oidc_uri(uri);
#[cfg(test)]
let bypass_proxy = bypass_proxy || dns_resolver_override.is_some();
#[cfg(test)]
let resolver: Arc<dyn reqwest::dns::Resolve> = dns_resolver_override.unwrap_or_else(|| Arc::new(resolver));
let mut builder = reqwest::Client::builder()
.dns_resolver(resolver)
.redirect(reqwest::redirect::Policy::none())
.timeout(OIDC_HTTP_REQUEST_TIMEOUT)
.connect_timeout(OIDC_HTTP_CONNECT_TIMEOUT);
if bypass_proxy {
builder = builder.no_proxy();
}
if !extra_root_certs.is_empty() {
builder = builder.tls_certs_merge(extra_root_certs.iter().cloned());
}
builder.build().map(|client| (client, url)).map_err(OidcHttpError::Reqwest)
}
fn oidc_forbidden_outbound_error(url: &Url, base: String, can_allow_origin: bool) -> OidcHttpError {
let reason = if can_allow_origin {
let origin = url.origin().ascii_serialization();
format!(
"{base}; add {origin} to {ENV_OUTBOUND_ALLOW_ORIGINS} (comma-separated) and restart RustFS to allow this operator-owned OIDC provider (origin only, no path)"
)
} else {
base
};
OidcHttpError::ForbiddenOutbound(reason)
}
fn oidc_http_error_from_reqwest(url: &Url, error: reqwest::Error) -> OidcHttpError {
if let Some(rejection) = find_outbound_dns_policy_rejection(&error) {
let base = rejection.to_string();
return oidc_forbidden_outbound_error(url, base, rejection.allow_origin_can_recover());
}
OidcHttpError::Reqwest(error)
}
/// Buffer a provider response body, failing closed once `limit` bytes have been seen.
///
/// `Response::bytes` would buffer the whole body unconditionally, so a hostile or compromised
/// provider endpoint could stream an arbitrarily large (or endless) body into memory.
async fn read_bounded_response_body(response: reqwest::Response, limit: usize) -> Result<Vec<u8>, OidcHttpError> {
if response.content_length().is_some_and(|len| len > limit as u64) {
return Err(OidcHttpError::ResponseTooLarge(limit));
}
let mut response = response;
let mut body = Vec::new();
while let Some(chunk) = response.chunk().await.map_err(OidcHttpError::Reqwest)? {
if body.len() + chunk.len() > limit {
return Err(OidcHttpError::ResponseTooLarge(limit));
}
body.extend_from_slice(&chunk);
}
Ok(body)
}
pub(super) fn should_bypass_proxy_for_oidc_uri(uri: &str) -> bool {
let Some(host) = Url::parse(uri).ok().and_then(|url| url.host_str().map(str::to_owned)) else {
return false;
};
let host = host.trim_matches(['[', ']']);
host.eq_ignore_ascii_case("localhost") || host.parse::<IpAddr>().is_ok_and(|addr| addr.is_loopback())
}
impl ReqwestHttpClient {
pub(super) fn new() -> Result<Self, String> {
Self::new_with_extra_root_certs(Vec::new())
}
fn extra_root_cert_cache(certs: Vec<Certificate>) -> Arc<RwLock<CachedOidcExtraRootCerts>> {
Arc::new(RwLock::new(CachedOidcExtraRootCerts {
generation: 0,
initialized: true,
certs,
}))
}
pub(super) fn new_with_extra_root_certs(extra_root_certs: Vec<Certificate>) -> Result<Self, String> {
Ok(Self {
policy_override: None,
extra_root_certs: Self::extra_root_cert_cache(extra_root_certs),
extra_root_ca_provider: None,
#[cfg(test)]
dns_resolver_override: None,
})
}
pub(super) fn new_with_extra_root_ca_provider(extra_root_ca_provider: OidcExtraRootCaProvider) -> Result<Self, String> {
Ok(Self {
policy_override: None,
extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())),
extra_root_ca_provider: Some(extra_root_ca_provider),
#[cfg(test)]
dns_resolver_override: None,
})
}
pub(super) async fn current_extra_root_certs(&self) -> Result<Vec<Certificate>, OidcHttpError> {
let Some(provider) = self.extra_root_ca_provider.as_ref() else {
return self
.extra_root_certs
.read()
.map(|cache| cache.certs.clone())
.map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")));
};
let material = provider.load().await.map_err(OidcHttpError::ExtraRootCa)?;
if let Ok(cache) = self.extra_root_certs.read()
&& cache.initialized
&& cache.generation == material.generation
{
return Ok(cache.certs.clone());
}
let certs = oidc_extra_root_certs(material.root_ca_pem.as_deref()).map_err(OidcHttpError::ExtraRootCa)?;
let mut cache = self
.extra_root_certs
.write()
.map_err(|e| OidcHttpError::ExtraRootCa(format!("extra root certificate cache lock poisoned: {e}")))?;
cache.generation = material.generation;
cache.initialized = true;
cache.certs = certs.clone();
Ok(certs)
}
/// Test-only constructor that pins outbound requests to an explicit policy, so a
/// loopback mock server can be reached without depending on process-wide environment.
#[cfg(test)]
pub(super) fn with_policy(policy: OutboundPolicy) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(Vec::new()),
extra_root_ca_provider: None,
dns_resolver_override: None,
}
}
#[cfg(test)]
pub(super) fn with_policy_and_extra_root_certs(policy: OutboundPolicy, extra_root_certs: Vec<Certificate>) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(extra_root_certs),
extra_root_ca_provider: None,
dns_resolver_override: None,
}
}
#[cfg(test)]
pub(super) fn with_policy_and_extra_root_ca_provider(
policy: OutboundPolicy,
extra_root_ca_provider: OidcExtraRootCaProvider,
) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Arc::new(RwLock::new(CachedOidcExtraRootCerts::default())),
extra_root_ca_provider: Some(extra_root_ca_provider),
dns_resolver_override: None,
}
}
#[cfg(test)]
pub(super) fn with_policy_and_dns_resolver(policy: OutboundPolicy, resolver: Arc<dyn reqwest::dns::Resolve>) -> Self {
Self {
policy_override: Some(policy),
extra_root_certs: Self::extra_root_cert_cache(Vec::new()),
extra_root_ca_provider: None,
dns_resolver_override: Some(resolver),
}
}
}
impl<'c> AsyncHttpClient<'c> for ReqwestHttpClient {
type Error = OidcHttpError;
type Future = Pin<Box<dyn Future<Output = Result<http::Response<Vec<u8>>, Self::Error>> + Send + 'c>>;
fn call(&'c self, request: http::Request<Vec<u8>>) -> Self::Future {
Box::pin(async move {
let started_at = Instant::now();
let (parts, body) = request.into_parts();
let method = parts.method.clone();
let uri = parts.uri.to_string();
if tracing::enabled!(tracing::Level::DEBUG) {
let request_headers = format_http_headers(&parts.headers);
debug!(
event = EVENT_OIDC_HTTP,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "request",
method = %method,
uri = %uri,
request_headers = %request_headers,
request_body_len = body.len(),
"oidc outbound http"
);
}
let extra_root_certs = self.current_extra_root_certs().await?;
let (client, url) = build_oidc_http_client(
&uri,
self.policy_override.as_ref(),
&extra_root_certs,
#[cfg(test)]
self.dns_resolver_override.clone(),
)?;
let response = client
.request(parts.method, uri.clone())
.headers(parts.headers)
.body(body)
.send()
.await;
let elapsed_ms = started_at.elapsed().as_millis().min(u128::from(u64::MAX)) as u64;
let succeeded = response.as_ref().is_ok_and(|resp| resp.status().is_success());
OIDC_PLUGIN_AUTHN_METRICS.record(elapsed_ms, succeeded);
let response = response.map_err(|err| {
let error = oidc_http_error_from_reqwest(&url, err);
error!(
event = EVENT_OIDC_HTTP,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "request_failed",
method = %method,
uri = %uri,
elapsed_ms,
error = %error,
"oidc outbound http"
);
error
})?;
let status = response.status();
let headers = response.headers().clone();
let body_bytes = read_bounded_response_body(response, MAX_OIDC_RESPONSE_SIZE)
.await
.map_err(|err| {
error!(
event = EVENT_OIDC_HTTP,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "response_body_failed",
method = %method,
uri = %uri,
status = status.as_u16(),
elapsed_ms,
error = %err,
"oidc outbound http"
);
err
})?;
if tracing::enabled!(tracing::Level::DEBUG) {
let response_headers = format_http_headers(&headers);
debug!(
event = EVENT_OIDC_HTTP,
component = LOG_COMPONENT_IAM,
subsystem = LOG_SUBSYSTEM_OIDC,
result = "response",
method = %method,
uri = %uri,
status = status.as_u16(),
status_success = status.is_success(),
elapsed_ms,
response_headers = %response_headers,
response_body_len = body_bytes.len(),
"oidc outbound http"
);
}
let mut http_response = http::Response::builder()
.status(status)
.body(body_bytes)
.map_err(OidcHttpError::Http)?;
*http_response.headers_mut() = headers;
Ok(http_response)
})
}
}
#[cfg(test)]
#[path = "transport_tests.rs"]
mod tests;
crates/iam/src/oidc/transport_tests.rs
+167
View File
@@ -0,0 +1,167 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::*;
use crate::oidc::test_support::*;
use openidconnect::AsyncHttpClient;
use rustfs_config::MAX_OIDC_RESPONSE_SIZE;
use rustfs_utils::egress::{ENV_OUTBOUND_ALLOW_ORIGINS, OutboundPolicy};
use std::sync::Arc;
#[test]
fn format_http_headers_redacts_sensitive_values() {
let mut headers = http::HeaderMap::new();
headers.insert(http::header::AUTHORIZATION, "Basic Y2xpZW50OnNlY3JldA==".parse().unwrap());
headers.insert(http::header::CONTENT_TYPE, "application/json".parse().unwrap());
headers.insert(http::header::COOKIE, "session=super-secret".parse().unwrap());
let rendered = format_http_headers(&headers);
// Sensitive header values never appear; only their length is emitted.
assert!(!rendered.contains("Y2xpZW50OnNlY3JldA=="), "authorization value leaked: {rendered}");
assert!(!rendered.contains("super-secret"), "cookie value leaked: {rendered}");
assert!(
rendered.contains("authorization=<redacted len="),
"expected redacted authorization: {rendered}"
);
assert!(rendered.contains("cookie=<redacted len="), "expected redacted cookie: {rendered}");
// Non-sensitive header values are preserved for diagnostics.
assert!(
rendered.contains("content-type=application/json"),
"content-type should be visible: {rendered}"
);
}
#[test]
fn is_sensitive_header_is_case_insensitive() {
assert!(is_sensitive_header("Authorization"));
assert!(is_sensitive_header("PROXY-AUTHORIZATION"));
assert!(is_sensitive_header("Set-Cookie"));
assert!(!is_sensitive_header("content-type"));
assert!(!is_sensitive_header("x-request-id"));
}
#[test]
fn build_oidc_http_client_rejects_forbidden_targets_without_allowlist() {
// Cloud metadata endpoint is never allowed.
assert!(
matches!(
build_oidc_http_client("http://169.254.169.254/latest/meta-data/", None, &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"metadata endpoint must be rejected"
);
// Loopback is rejected by default (no allow-origins configured).
assert!(
matches!(
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", None, &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"loopback must be rejected by default"
);
// A public hostname passes the up-front shape/host check; the resolved IP is still
// re-classified at connection time by the pinned resolver.
assert!(
build_oidc_http_client("https://accounts.example.com/.well-known/openid-configuration", None, &[], None).is_ok(),
"public https endpoint should build"
);
}
#[test]
fn test_should_bypass_proxy_for_oidc_uri_loopback_only() {
assert!(should_bypass_proxy_for_oidc_uri("http://127.0.0.1:9000/.well-known/openid-configuration"));
assert!(should_bypass_proxy_for_oidc_uri("http://localhost:9000/.well-known/openid-configuration"));
assert!(should_bypass_proxy_for_oidc_uri("http://[::1]:9000/.well-known/openid-configuration"));
assert!(!should_bypass_proxy_for_oidc_uri(
"https://idp.example.com/.well-known/openid-configuration"
));
assert!(!should_bypass_proxy_for_oidc_uri("not-a-url"));
}
#[test]
fn build_oidc_http_client_honors_explicit_allowlist_for_loopback() {
let policy = OutboundPolicy::from_allowed_origins("http://127.0.0.1:8080").expect("origin should parse");
assert!(
build_oidc_http_client("http://127.0.0.1:8080/.well-known/openid-configuration", Some(&policy), &[], None).is_ok(),
"explicitly allow-listed loopback origin should build"
);
// A metadata endpoint stays forbidden even when a loopback origin is allow-listed.
assert!(
matches!(
build_oidc_http_client("http://169.254.169.254/", Some(&policy), &[], None),
Err(OidcHttpError::ForbiddenOutbound(_))
),
"metadata endpoint stays forbidden despite an unrelated allow-list entry"
);
}
#[tokio::test]
async fn oidc_nonrecoverable_dns_policy_rejection_omits_allowlist_hint() {
let uri = "http://metadata.internal/latest/meta-data";
let http_client = ReqwestHttpClient::with_policy_and_dns_resolver(
OutboundPolicy::default(),
Arc::new(RejectingDnsResolver {
allow_origin_can_recover: false,
calls: None,
}),
);
let request = http::Request::builder()
.uri(uri)
.body(Vec::new())
.expect("request should build");
let error = http_client
.call(request)
.await
.expect_err("metadata DNS answer should be rejected");
let message = error.to_string();
assert!(matches!(error, OidcHttpError::ForbiddenOutbound(_)));
assert!(message.contains("metadata.internal"));
assert!(!message.contains(&format!("add http://metadata.internal to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
}
#[test]
fn oidc_metadata_endpoint_rejection_does_not_offer_allowlist_bypass() {
let error = build_oidc_http_client("http://169.254.169.254/latest/meta-data/", Some(&OutboundPolicy::default()), &[], None)
.expect_err("metadata endpoint must remain forbidden");
let message = error.to_string();
assert!(message.contains("metadata endpoint"));
assert!(!message.contains(&format!("add http://169.254.169.254 to {ENV_OUTBOUND_ALLOW_ORIGINS}")));
}
#[tokio::test]
async fn oidc_response_body_at_the_limit_is_accepted() {
let Some((base, handle)) = start_unbounded_body_server(MAX_OIDC_RESPONSE_SIZE) else {
return;
};
let body = fetch_oidc_mock_body(&base)
.await
.expect("a body at the limit must be accepted");
assert_eq!(body.len(), MAX_OIDC_RESPONSE_SIZE);
handle.join().expect("mock body server thread should exit");
}
#[tokio::test]
async fn oidc_response_body_past_the_limit_is_rejected() {
let Some((base, handle)) = start_unbounded_body_server(MAX_OIDC_RESPONSE_SIZE + 1) else {
return;
};
let err = fetch_oidc_mock_body(&base)
.await
.map(|body| body.len())
.expect_err("an oversized provider response must fail closed instead of being buffered");
assert!(
matches!(err, OidcHttpError::ResponseTooLarge(MAX_OIDC_RESPONSE_SIZE)),
"unexpected error: {err}"
);
handle.join().expect("mock body server thread should exit");
}
crates/log-analyzer/src/rules/seed/heal.rs
+20 -5
View File
@@ -15,7 +15,7 @@
//! Self-heal rules.
use super::super::model::{Rule, Severity::*};
use super::{any, base, contains, prefix, strings};
use super::{all, any, base, contains, field, prefix, strings};
pub(super) fn rules() -> Vec<Rule> {
vec![
@@ -33,14 +33,29 @@ pub(super) fn rules() -> Vec<Rule> {
)
},
Rule {
anchors: strings(["all drives had write errors, unable to heal"]),
evidence_fields: strings([
"bucket",
"object",
"version_id",
"target_count",
"writer_failure_count",
"error",
]),
anchors: strings(["Set disk object heal writer failures"]),
..base(
"heal-all-writes-failed",
P0DataRisk,
"heal",
"heal 结果无法写入任何盘",
contains("all drives had write errors, unable to heal"),
"heal 结果无法写入任何盘(全盘故障/满盘)。",
"所有自愈目标不可写",
any([
contains("all drives had write errors, unable to heal"),
all([
contains("Set disk object heal writer failures"),
field("state", "writer_unavailable"),
field("result", "all_targets_unavailable"),
]),
]),
"所有自愈目标不可写,修复无法继续;健康源盘可能仍可读。",
"先解决容量/盘故障 finding,再重跑 heal。",
)
},
crates/log-analyzer/src/rules/seed/tests.rs
+40
View File
@@ -297,6 +297,46 @@ fn every_rule_has_a_positive_sample() {
}
}
#[test]
fn heal_writer_rule_requires_all_targets_and_retains_legacy_messages() {
let engine = engine();
for (sample, expected) in [
(
Sample {
message: "Set disk object heal writer failures",
level: Some(LogLevel::Warn),
fields: &[("state", "writer_unavailable"), ("result", "all_targets_unavailable")],
..Default::default()
},
true,
),
(
Sample {
message: "Set disk object heal writer failures",
level: Some(LogLevel::Warn),
fields: &[("state", "writer_unavailable"), ("result", "partial_targets_unavailable")],
..Default::default()
},
false,
),
(
Sample {
message: "Set disk object heal writer failures",
fields: &[("result", "all_targets_unavailable")],
..Default::default()
},
false,
),
(msg("all drives had write errors, unable to heal b/o"), true),
] {
let matched = engine
.matches(&event(&sample))
.iter()
.any(|&index| engine.rules()[index].id == "heal-all-writes-failed");
assert_eq!(matched, expected);
}
}
#[test]
fn smoke_samples_hit_exact_rule_sets() {
let engine = engine();
crates/obs/src/config.rs
+52 -5
View File
@@ -238,8 +238,10 @@ impl OtelConfig {
/// that know their display/build version should use
/// [`OtelConfig::extract_otel_config_from_env_with_version`].
///
/// When no endpoint is configured at all, `use_stdout` is forced to `true`
/// so that logs are still visible during development.
/// When no endpoint is configured at all (neither the root endpoint nor any
/// per-signal one), `use_stdout` is forced to `true` so that logs are still
/// visible during development. With any endpoint set, `RUSTFS_OBS_USE_STDOUT`
/// is honoured.
///
/// # Example
/// ```no_run
@@ -268,9 +270,15 @@ impl OtelConfig {
_ => env::var(ENV_OBS_ENDPOINT).unwrap_or_default(),
};
// Force stdout when there is no remote endpoint so that operators
// always have *some* log output in the default configuration.
let use_stdout = if endpoint.is_empty() {
// Force stdout only when no OTLP endpoint is configured at all, so that operators
// always have *some* log output in the default configuration. A per-signal endpoint
// (e.g. `RUSTFS_OBS_TRACE_ENDPOINT`) counts as configured: otherwise setting only that
// one forces the stdout exporters on and silently ignores `RUSTFS_OBS_USE_STDOUT=false`,
// printing every exported span to stdout.
let has_signal_endpoint = [ENV_OBS_TRACE_ENDPOINT, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_LOG_ENDPOINT]
.into_iter()
.any(|key| get_env_opt_str(key).is_some_and(|value| !value.is_empty()));
let use_stdout = if endpoint.is_empty() && !has_signal_endpoint {
true
} else {
get_env_bool(ENV_OBS_USE_STDOUT, USE_STDOUT)
@@ -538,6 +546,45 @@ mod tests {
});
}
#[test]
fn use_stdout_is_forced_only_when_no_endpoint_is_configured() {
with_profiling_env_lock(|| {
temp_env::with_vars_unset(
[
ENV_OBS_ENDPOINT,
ENV_OBS_TRACE_ENDPOINT,
ENV_OBS_METRIC_ENDPOINT,
ENV_OBS_LOG_ENDPOINT,
],
|| {
temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || {
// No endpoint at all: forced on, the explicit false is ignored.
assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true));
});
for key in [ENV_OBS_TRACE_ENDPOINT, ENV_OBS_METRIC_ENDPOINT, ENV_OBS_LOG_ENDPOINT] {
temp_env::with_var(key, Some("http://localhost:4318/v1/signal"), || {
temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || {
assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(false), "{key}");
});
temp_env::with_var(ENV_OBS_USE_STDOUT, Some("true"), || {
assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true), "{key}");
});
temp_env::with_var_unset(ENV_OBS_USE_STDOUT, || {
assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(USE_STDOUT), "{key}");
});
});
}
temp_env::with_var(ENV_OBS_TRACE_ENDPOINT, Some(""), || {
temp_env::with_var(ENV_OBS_USE_STDOUT, Some("false"), || {
// An empty per-signal endpoint is not a configured endpoint.
assert_eq!(OtelConfig::extract_otel_config_from_env(None).use_stdout, Some(true));
});
});
},
);
});
}
#[test]
fn stdout_mirror_environment_preserves_unset_true_and_false() {
with_profiling_env_lock(|| {
crates/policy/src/policy/variables.rs
+89 -19
View File
@@ -252,6 +252,13 @@ impl PolicyVariableResolver for VariableResolver {
}
pub async fn resolve_aws_variables(pattern: &str, resolver: &dyn PolicyVariableResolver) -> Vec<String> {
resolve_aws_variables_with_depth(pattern, resolver, 0).await
}
async fn resolve_aws_variables_with_depth(pattern: &str, resolver: &dyn PolicyVariableResolver, depth: usize) -> Vec<String> {
if depth >= 10 {
return vec![pattern.to_string()];
}
let mut results = vec![pattern.to_string()];
let mut changed = true;
@@ -264,7 +271,7 @@ pub async fn resolve_aws_variables(pattern: &str, resolver: &dyn PolicyVariableR
let mut new_results = Vec::new();
for result in &results {
let resolved = resolve_single_pass(result, resolver).await;
let resolved = resolve_single_pass(result, resolver, depth).await;
if resolved.len() > 1 || (resolved.len() == 1 && &resolved[0] != result) {
changed = true;
}
@@ -288,30 +295,31 @@ pub async fn resolve_aws_variables(pattern: &str, resolver: &dyn PolicyVariableR
fn resolve_aws_variables_boxed<'a>(
pattern: &'a str,
resolver: &'a dyn PolicyVariableResolver,
depth: usize,
) -> std::pin::Pin<Box<dyn Future<Output = Vec<String>> + Send + 'a>> {
Box::pin(resolve_aws_variables(pattern, resolver))
Box::pin(resolve_aws_variables_with_depth(pattern, resolver, depth))
}
/// Single pass resolution of variables in a string
async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolver) -> Vec<String> {
async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolver, depth: usize) -> Vec<String> {
// Find all ${...} format variables
let mut results = vec![pattern.to_string()];
let mut results = vec![(pattern.to_string(), 0)];
// Process each result string
let mut i = 0;
while i < results.len() {
let mut start = 0;
let mut start = results[i].1;
let mut modified = false;
// Find variables in current string
while let Some(pos) = results[i][start..].find("${") {
while let Some(pos) = results[i].0[start..].find("${") {
let actual_pos = start + pos;
// Find the matching closing brace, taking into account nested braces
let mut brace_count = 1;
let mut end_pos = actual_pos + 2; // Start after "${"
let bytes = results[i].as_bytes();
let bytes = results[i].0.as_bytes();
while end_pos < bytes.len() && brace_count > 0 {
match bytes[end_pos] {
b'{' => brace_count += 1,
@@ -324,19 +332,23 @@ async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolve
}
if brace_count == 0 {
let var_name = &results[i][actual_pos + 2..end_pos];
let var_name = &results[i].0[actual_pos + 2..end_pos];
// Check if this is a nested variable (contains ${...} inside)
if var_name.contains("${") {
// For nested variables like ${${a}-${b}}, we need to resolve the inner variables first
// Then use the resolved result as a new variable to resolve
let resolved_inner = resolve_aws_variables_boxed(var_name, resolver).await;
let resolved_inner = resolve_aws_variables_boxed(var_name, resolver, depth + 1).await;
if resolved_inner.len() == 1 && resolved_inner[0] == var_name {
start = end_pos + 1;
continue;
}
let mut new_results = Vec::new();
for resolved_var_name in resolved_inner {
let prefix = &results[i][..actual_pos];
let suffix = &results[i][end_pos + 1..];
new_results.push(format!("{prefix}{resolved_var_name}{suffix}"));
let prefix = &results[i].0[..actual_pos];
let suffix = &results[i].0[end_pos + 1..];
new_results.push((format!("{prefix}{resolved_var_name}{suffix}"), actual_pos + resolved_var_name.len()));
}
if !new_results.is_empty() {
@@ -354,11 +366,13 @@ async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolve
if !values.is_empty() {
// If there are multiple values, create a new result for each value
let mut new_results = Vec::new();
let prefix = &results[i][..actual_pos];
let suffix = &results[i][end_pos + 1..];
let prefix = &results[i].0[..actual_pos];
let suffix = &results[i].0[end_pos + 1..];
for value in values {
new_results.push(format!("{prefix}{value}{suffix}"));
// Resume after the substitution; a generated placeholder
// belongs to the next bounded pass, not this one.
new_results.push((format!("{prefix}{value}{suffix}"), actual_pos + value.len()));
}
results.splice(i..i + 1, new_results);
@@ -367,9 +381,9 @@ async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolve
} else {
// Variable resolved to empty, just remove the variable placeholder
let mut new_results = Vec::new();
let prefix = &results[i][..actual_pos];
let suffix = &results[i][end_pos + 1..];
new_results.push(format!("{prefix}{suffix}"));
let prefix = &results[i].0[..actual_pos];
let suffix = &results[i].0[end_pos + 1..];
new_results.push((format!("{prefix}{suffix}"), actual_pos));
results.splice(i..i + 1, new_results);
modified = true;
@@ -391,7 +405,7 @@ async fn resolve_single_pass(pattern: &str, resolver: &dyn PolicyVariableResolve
}
}
results
results.into_iter().map(|(value, _)| value).collect()
}
#[cfg(test)]
@@ -399,6 +413,62 @@ mod tests {
use super::*;
use serde_json::Value;
use std::collections::HashMap;
use std::sync::atomic::{AtomicUsize, Ordering};
struct CycleResolver {
calls: AtomicUsize,
}
#[async_trait]
impl PolicyVariableResolver for CycleResolver {
async fn resolve(&self, name: &str) -> Option<String> {
// Make the vulnerable implementation terminate so a regression
// reports an assertion failure instead of hanging the test runner.
if self.calls.fetch_add(1, Ordering::Relaxed) >= 100 {
return None;
}
match name {
"aws:AccountId" => Some("$$".to_string()),
"aws:username" => Some("{aws:AccountId}{aws:username}".to_string()),
_ => None,
}
}
fn is_dynamic(&self, _: &str) -> bool {
false
}
}
#[tokio::test]
async fn generated_policy_variable_cycles_are_bounded() {
let resolver = CycleResolver {
calls: AtomicUsize::new(0),
};
let result = resolve_aws_variables("${aws:AccountId}{aws:username}", &resolver).await;
assert!(resolver.calls.load(Ordering::Relaxed) <= 10);
assert_eq!(result, vec!["${aws:AccountId}{aws:username}".to_string()]);
}
#[tokio::test]
async fn a_single_pass_resolves_all_original_variables() {
let mut context = VariableContext::new();
context.username = Some("alice".to_string());
let resolver = VariableResolver::new(context);
let pattern = "${aws:username}".repeat(20);
assert_eq!(resolve_single_pass(&pattern, &resolver, 0).await, vec!["alice".repeat(20)]);
}
#[tokio::test]
async fn nested_policy_variables_have_a_depth_limit() {
let resolver = CycleResolver {
calls: AtomicUsize::new(0),
};
let pattern = format!("{}aws:username{}", "${".repeat(100), "}".repeat(100));
let result = resolve_aws_variables(&pattern, &resolver).await;
assert_eq!(result.len(), 1);
assert!(result[0].contains("${"));
assert!(resolver.calls.load(Ordering::Relaxed) <= 10);
}
#[tokio::test]
async fn test_resolve_aws_variables_with_username() {
crates/rio/Cargo.toml
+1
View File
@@ -80,6 +80,7 @@ rustfs-tls-runtime.workspace = true
rustfs-utils = { workspace = true, features = ["io", "hash", "compress"] }
serde_json = { workspace = true, features = ["raw_value"] }
md-5 = { workspace = true }
hmac.workspace = true
minlz.workspace = true
tracing.workspace = true
thiserror.workspace = true
crates/rio/src/etag_reader.rs
+64
View File
@@ -14,8 +14,10 @@
use crate::compress_index::{Index, TryGetIndex};
use crate::{BadDigest, EtagResolvable, HashReaderDetector, HashReaderMut};
use hmac::{Hmac, KeyInit, Mac};
use md5::{Digest, Md5};
use pin_project_lite::pin_project;
use sha2::Sha256;
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
@@ -29,6 +31,7 @@ pin_project! {
pub finished: bool,
pub checksum: Option<String>,
resolved_etag: Option<String>,
encrypted_etag_mac: Option<Hmac<Sha256>>,
}
}
@@ -40,9 +43,21 @@ impl<R> EtagReader<R> {
finished: false,
checksum,
resolved_etag: None,
encrypted_etag_mac: None,
}
}
/// Protect the content fingerprint with the object's encryption material.
/// Plaintext checksum verification remains the inner reader's responsibility.
pub fn new_encrypted(inner: R, key: [u8; 32], nonce: [u8; 12]) -> std::io::Result<Self> {
let mut mac = <Hmac<Sha256> as KeyInit>::new_from_slice(&key).map_err(std::io::Error::other)?;
mac.update(b"RustFS:encrypted-ETag:v1\0");
mac.update(&nonce);
let mut reader = Self::new(inner, None);
reader.encrypted_etag_mac = Some(mac);
Ok(reader)
}
/// Get the final md5 value (etag) as a hex string, only compute once.
/// Can be called multiple times, always returns the same result after finished.
pub fn get_etag(&mut self) -> String {
@@ -108,6 +123,14 @@ impl<R> EtagResolvable for EtagReader<R> {
true
}
fn try_resolve_etag(&mut self) -> Option<String> {
if let Some(mut mac) = self.encrypted_etag_mac.clone() {
if !self.finished {
return None;
}
mac.update(self.get_etag().as_bytes());
let protected = mac.finalize().into_bytes();
return Some(hex_simd::encode_to_string(&protected[..16], hex_simd::AsciiCase::Lower));
}
// EtagReader provides its own etag, not delegating to inner
if let Some(checksum) = &self.checksum {
Some(checksum.clone())
@@ -148,6 +171,47 @@ mod tests {
use std::io::Cursor;
use tokio::io::{AsyncReadExt, BufReader};
#[tokio::test]
async fn encrypted_etags_hide_plaintext_fingerprints_including_empty_objects() {
for data in [b"".as_slice(), b"Salary 2026: 85000 EUR".as_slice()] {
let md5 = hex_simd::encode_to_string(Md5::digest(data), hex_simd::AsciiCase::Lower);
let mut tags = Vec::new();
for (key, nonce) in [([1; 32], [1; 12]), ([2; 32], [1; 12]), ([1; 32], [2; 12])] {
let mut reader = EtagReader::new_encrypted(Cursor::new(data), key, nonce).expect("create protected ETag reader");
assert!(reader.try_resolve_etag().is_none());
let mut output = Vec::new();
reader.read_to_end(&mut output).await.expect("read unchanged plaintext");
assert_eq!(output, data);
let tag = reader.try_resolve_etag().expect("resolve protected ETag after EOF");
assert_eq!(tag.len(), 32);
assert_ne!(tag, md5);
assert_eq!(reader.try_resolve_etag().as_ref(), Some(&tag));
tags.push(tag);
}
assert_ne!(tags[0], tags[1], "different keys must not link identical content");
assert_ne!(tags[0], tags[2], "different objects under a legacy key must not link identical content");
}
}
#[tokio::test]
async fn encrypted_etag_reader_preserves_plaintext_digest_rejection() {
let plaintext = crate::HashReader::from_stream(
Cursor::new(b"actual plaintext"),
16,
16,
Some("00000000000000000000000000000000".to_owned()),
None,
false,
)
.expect("create checked plaintext reader");
let mut reader = EtagReader::new_encrypted(plaintext, [1; 32], [2; 12]).expect("create protected ETag reader");
let err = reader
.read_to_end(&mut Vec::new())
.await
.expect_err("wrong plaintext MD5 must still be rejected");
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
}
#[tokio::test]
async fn test_etag_reader_basic() {
let data = b"hello world";
crates/rio/src/hash_reader.rs
+5
View File
@@ -136,6 +136,7 @@ pin_project! {
content_sha256: Option<String>,
content_sha256_hasher: Option<Sha256Hasher>,
checksum_on_finish: bool,
pub encrypted_etag: bool,
// Read only at EOF; see the timing contract in `crate::trailer`.
trailer: Option<SharedTrailerSource>,
@@ -184,6 +185,7 @@ impl HashReader {
content_sha256: sha256hex.clone(),
content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()),
checksum_on_finish: false,
encrypted_etag: false,
trailer: None,
})
}
@@ -224,6 +226,7 @@ impl HashReader {
content_sha256: sha256hex.clone(),
content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()),
checksum_on_finish: false,
encrypted_etag: false,
trailer: None,
})
}
@@ -294,6 +297,7 @@ impl HashReader {
content_hash,
content_hasher,
checksum_on_finish: false,
encrypted_etag: false,
trailer: existing_hash_reader.get_trailer().cloned(),
})
} else {
@@ -322,6 +326,7 @@ impl HashReader {
content_sha256: sha256hex.clone(),
content_sha256_hasher: sha256hex.map(|_| Sha256Hasher::new()),
checksum_on_finish: false,
encrypted_etag: false,
trailer: None,
})
}
crates/scanner-metrics/src/metrics.rs
+85 -1
View File
@@ -18,7 +18,7 @@ use jiff::Timestamp;
use rustfs_heal_contracts::heal_channel::HealScanMode;
use serde::{Deserialize, Serialize};
use std::{
collections::{BTreeSet, HashMap},
collections::{BTreeMap, BTreeSet, HashMap},
fmt::Display,
future::Future,
pin::Pin,
@@ -994,6 +994,8 @@ pub struct Metrics {
scanner_bitrot_cycle_enabled: AtomicBool,
scanner_bitrot_cycle_millis: AtomicU64,
scanner_checkpoint: Mutex<Option<ScannerCheckpointReport>>,
scanner_checkpoints_persisted: Mutex<BTreeMap<(usize, usize), ScannerPersistedCheckpointReport>>,
scanner_checkpoint_save_failures: AtomicU64,
scanner_checkpoint_used: AtomicU64,
scanner_checkpoint_cleared: AtomicU64,
scanner_checkpoint_ignored: AtomicU64,
@@ -1137,6 +1139,15 @@ pub struct ScannerCheckpointReport {
pub last_event: String,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct ScannerPersistedCheckpointReport {
pub pool_index: usize,
pub set_index: usize,
pub bucket: String,
pub saved_unix_secs: u64,
pub checkpoint: Option<ScannerCheckpointReport>,
}
#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
pub struct ScannerSourceWorkSnapshot {
pub source: String,
@@ -1499,6 +1510,10 @@ pub struct ScannerMetricsReport {
#[serde(skip_serializing_if = "Option::is_none")]
pub scan_checkpoint: Option<ScannerCheckpointReport>,
#[serde(default)]
pub scan_checkpoints_persisted: Vec<ScannerPersistedCheckpointReport>,
#[serde(default)]
pub scan_checkpoint_save_failures: u64,
#[serde(default)]
pub scan_checkpoint_used: u64,
#[serde(default)]
pub scan_checkpoint_cleared: u64,
@@ -2046,6 +2061,8 @@ impl Metrics {
scanner_bitrot_cycle_enabled: AtomicBool::new(false),
scanner_bitrot_cycle_millis: AtomicU64::new(0),
scanner_checkpoint: Mutex::new(None),
scanner_checkpoints_persisted: Mutex::new(BTreeMap::new()),
scanner_checkpoint_save_failures: AtomicU64::new(0),
scanner_checkpoint_used: AtomicU64::new(0),
scanner_checkpoint_cleared: AtomicU64::new(0),
scanner_checkpoint_ignored: AtomicU64::new(0),
@@ -2353,6 +2370,34 @@ impl Metrics {
}
}
/// Keep only the latest confirmed periodic save per set on this process.
pub fn record_scanner_checkpoint_saved(
&self,
pool_index: usize,
set_index: usize,
bucket: &str,
checkpoint: Option<ScannerCheckpointReport>,
) {
let mut saved = self
.scanner_checkpoints_persisted
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
saved.insert(
(pool_index, set_index),
ScannerPersistedCheckpointReport {
pool_index,
set_index,
bucket: bucket.to_string(),
saved_unix_secs: unix_now_secs(),
checkpoint,
},
);
}
pub fn record_scanner_checkpoint_save_failed(&self) {
self.scanner_checkpoint_save_failures.fetch_add(1, Ordering::Relaxed);
}
pub fn record_scanner_checkpoint_used(&self) {
self.scanner_checkpoint_used.fetch_add(1, Ordering::Relaxed);
self.update_scanner_checkpoint_event(SCANNER_CHECKPOINT_EVENT_USED);
@@ -3512,6 +3557,14 @@ impl Metrics {
Ok(checkpoint) => checkpoint.clone(),
Err(poisoned) => poisoned.into_inner().clone(),
};
m.scan_checkpoints_persisted = self
.scanner_checkpoints_persisted
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.values()
.cloned()
.collect();
m.scan_checkpoint_save_failures = self.scanner_checkpoint_save_failures.load(Ordering::Relaxed);
m.scan_checkpoint_used = self.scanner_checkpoint_used.load(Ordering::Relaxed);
m.scan_checkpoint_cleared = self.scanner_checkpoint_cleared.load(Ordering::Relaxed);
m.scan_checkpoint_ignored = self.scanner_checkpoint_ignored.load(Ordering::Relaxed);
@@ -3975,6 +4028,37 @@ mod tests {
assert_eq!(report.scan_checkpoint_cleared, 1);
}
#[tokio::test]
async fn report_distinguishes_persisted_checkpoints_from_memory_and_failures() {
let metrics = Metrics::new();
let saved = ScannerCheckpointReport {
version: 1,
resume_after: "bucket/first".to_string(),
reason: "objects".to_string(),
last_event: "saved".to_string(),
};
metrics.record_scanner_checkpoint_saved(0, 1, "bucket", Some(saved.clone()));
metrics.record_scanner_checkpoint_set(1, "bucket/second", "objects");
metrics.record_scanner_checkpoint_save_failed();
let report = metrics.report().await;
assert_eq!(report.scan_checkpoint_save_failures, 1);
assert_eq!(report.scan_checkpoint.expect("in-memory cursor").resume_after, "bucket/second");
assert_eq!(report.scan_checkpoints_persisted.len(), 1);
assert_eq!(report.scan_checkpoints_persisted[0].checkpoint, Some(saved));
assert_eq!(report.scan_checkpoints_persisted[0].set_index, 1);
assert!(report.scan_checkpoints_persisted[0].saved_unix_secs > 0);
metrics.record_scanner_checkpoint_saved(0, 1, "other-bucket", None);
metrics.record_scanner_checkpoint_saved(0, 2, "bucket", None);
let report = metrics.report().await;
assert_eq!(
report.scan_checkpoints_persisted.len(),
2,
"retain at most one save per set, not per bucket"
);
assert_eq!(report.scan_checkpoints_persisted[0].bucket, "other-bucket");
assert!(report.scan_checkpoints_persisted[0].checkpoint.is_none());
}
#[tokio::test]
async fn report_includes_scanner_source_work() {
let metrics = Metrics::new();
crates/scanner/src/data_usage_define.rs
+39 -3
View File
@@ -234,7 +234,7 @@ pub static DATA_USAGE_BLOOM_RECOVERY_PATH: LazyLock<String> =
pub static BACKGROUND_HEAL_INFO_PATH: LazyLock<String> =
LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}.background-heal.json"));
const MAX_DATA_USAGE_CACHE_DEPTH: usize = 1024;
pub(crate) const MAX_DATA_USAGE_CACHE_DEPTH: usize = 1024;
/// Scanner-side accounting on the shared [`SizeSummary`].
///
@@ -948,6 +948,42 @@ impl DataUsageCache {
return self.prepare_for_scan(name, next_cycle, leader_epoch, source, scan_plan_digest, true);
}
if !reusable {
let had_resume_state = self.info.scan_checkpoint.is_some()
|| self.info.scan_resume_after.is_some()
|| self.info.scan_coverage_receipt.is_some()
|| self.info.scan_raw_enumeration_cursor.is_some()
|| self.info.scan_raw_enumeration_page_index.is_some();
if had_resume_state {
let reason = if name == DATA_USAGE_ROOT {
"invalid_bucket_scope"
} else if !identity.is_valid() {
"invalid_identity"
} else if self.info.name != name {
"bucket_mismatch"
} else if self.info.source != Some(source) {
"source_mismatch"
} else if self.info.cache_key_format != DATA_USAGE_CACHE_KEY_FORMAT {
"key_format_mismatch"
} else if self.info.scan_identity != Some(identity)
|| self.info.tier_registry_generation != Some(identity.tier_registry_generation)
{
"identity_mismatch"
} else if self.info.leader_epoch != leader_epoch && !cross_epoch_checkpoint {
"unvalidated_handoff"
} else {
"incomplete_cache_scope"
};
warn!(
target: "rustfs::scanner::data_usage",
event = EVENT_SCANNER_CACHE_LOAD_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_CACHE,
state = "checkpoint_reset",
bucket = %name,
reason,
"Scanner bucket checkpoint cannot be adopted"
);
}
let keep_debts = self.info.name == name
&& self
.info
@@ -1221,8 +1257,8 @@ impl DataUsageCache {
self.cache.get(&hash_path(path).key())
}
pub fn find_children_copy(&mut self, h: DataUsageHash) -> DataUsageHashMap {
self.cache.entry(h.string()).or_default().children.clone()
pub fn find_children_copy(&self, h: &DataUsageHash) -> DataUsageHashMap {
self.cache.get(&h.0).map(|entry| entry.children.clone()).unwrap_or_default()
}
pub fn flatten(&self, root: &DataUsageEntry) -> DataUsageEntry {
crates/scanner/src/data_usage_define/tests.rs
+24 -4
View File
@@ -2491,12 +2491,32 @@ fn checked_flatten_accepts_depth_limit_and_rejects_deeper_tree() {
}
#[test]
fn test_find_children_copy_preserves_missing_entry_behavior() {
let mut cache = DataUsageCache::default();
fn test_find_children_copy_does_not_insert_missing_entries() {
let cache = DataUsageCache::default();
let missing_hash = hash_path("missing");
assert!(cache.find_children_copy(missing_hash.clone()).is_empty());
assert!(cache.cache.contains_key(&missing_hash.key()));
assert!(cache.find_children_copy(&missing_hash).is_empty());
assert!(cache.cache.is_empty());
assert_eq!(cache.cache.capacity(), 0);
}
#[test]
fn test_find_children_copy_preserves_existing_children() {
let hash = hash_path("bucket");
let children = HashSet::from(["bucket/child".to_string()]);
let mut cache = DataUsageCache::default();
cache.cache.insert(
hash.key(),
DataUsageEntry {
children: children.clone(),
..Default::default()
},
);
let mut copied = cache.find_children_copy(&hash);
assert_eq!(copied, children);
copied.clear();
assert_eq!(cache.cache[&hash.0].children, children);
assert_eq!(cache.cache.len(), 1);
}
#[test]
crates/scanner/src/remote_scanner/stream.rs
+7 -3
View File
@@ -853,7 +853,7 @@ async fn scan_and_persist_local_bucket(
)
.await
{
ScannerCheckpointPersistResult::Saved => {
ScannerCheckpointPersistResult::Saved | ScannerCheckpointPersistResult::Skipped => {
if guard.is_lock_lost() {
scan_ctx.cancel();
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
@@ -869,8 +869,12 @@ async fn scan_and_persist_local_bucket(
"remote namespace scanner cache fence changed during checkpoint save",
));
}
ScannerCheckpointPersistResult::Failed(_error) => {
checkpoint_channel_closed = true;
ScannerCheckpointPersistResult::RetryBucket(error) => {
scan_ctx.cancel();
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
return Err(RemoteScannerServerError::retry_bucket(format!(
"remote namespace scanner checkpoint needs reload: {error}",
)));
}
}
}
crates/scanner/src/scanner/backlog.rs
+89 -23
View File
@@ -2192,7 +2192,7 @@ mod tests {
})
.expect("native retirement test thread")
.join()
.expect("native retirement test completed");
.unwrap_or_else(|panic| std::panic::resume_unwind(panic));
}
async fn native_retirement_store() -> (tempfile::TempDir, Arc<ECStore>) {
@@ -2631,13 +2631,13 @@ mod tests {
)
.await;
eprintln!("native expansion fixture old_pools={old_pool_count} stage=seed_old_authority");
let fault = unstable_source
.then(|| NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&old.pools[0].disk_set[0]), "publish", 2));
let now = unix_now();
let mut controller = ScannerPauseBacklogController::claim(Arc::clone(&old), now)
.await
.expect("the native controller commits the actual old cohort");
if unstable_source {
let original = if unstable_source {
let fault =
NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&old.pools[0].disk_set[0]), "publish", 2);
let controller = ScannerPauseBacklogController::claim(Arc::clone(&old), now)
.await
.expect("the native controller commits the actual old cohort");
assert!(controller.loaded.requires_reload);
let (bytes, _) = native_replica_bytes(&old.pools[0].disk_set[0]).await;
let ScannerPauseBacklogReplicaState::Valid(record) = decode_scanner_pause_backlog_ledger(&bytes) else {
@@ -2645,23 +2645,14 @@ mod tests {
};
assert!(record.stable.is_none());
assert_eq!(record.committed.unwrap().ledger, controller.loaded.ledger);
let original = controller.loaded.ledger.clone();
assert!(controller.loaded.durable);
drop(controller);
drop(fault);
original
} else {
controller.observe(observation(now + 1, true, 4)).await;
controller.observe(observation(now + 2, false, 4)).await;
let decision = controller.begin_attempt(now + 2).await;
assert!(
matches!(decision, ScannerPauseBacklogAttemptDecision::Tracked(_)),
"expected tracked native expansion setup attempt, got {decision:?}; ledger={:?}; persistence_disabled={}; runtime_error={:?}",
controller.loaded.ledger,
controller.persistence_disabled,
runtime_error()
);
assert!(controller.loaded.ledger.has_unfinished_attempt());
}
let original = controller.loaded.ledger.clone();
assert!(controller.loaded.durable);
drop(controller);
drop(fault);
seed_native_expansion_ledger(&old, now).await
};
old.pool_meta_write_status()
.await
.expect("healthy pool metadata keeps the background recovery loop read-only");
@@ -2686,6 +2677,81 @@ mod tests {
(root, expanded, original)
}
async fn seed_native_expansion_ledger(store: &Arc<ECStore>, now: u64) -> ScannerPauseBacklogLedger {
let loaded = load_scanner_pause_backlog(Arc::clone(store))
.await
.expect("empty native expansion cohort");
assert!(
loaded
.replicas
.iter()
.all(|replica| matches!(replica.state, ScannerPauseBacklogReplicaState::Missing))
);
let mut ledger = claim_scanner_pause_backlog_writer(&loaded.ledger, now).expect("fixture writer generation");
// The fixture needs the final unfinished attempt, not four separate
// durable controller transitions before the fault under test. Retain
// their generations and timestamps, then publish the final state once
// through the real native commit and stabilization protocol.
ledger.apply_observation(observation(now + 1, true, 4));
prepare_scanner_pause_backlog_persist(&mut ledger, now + 1).expect("fixture pause generation");
ledger.apply_observation(observation(now + 2, false, 4));
prepare_scanner_pause_backlog_persist(&mut ledger, now + 2).expect("fixture resume generation");
assert!(matches!(ledger.begin_attempt(now + 2), ScannerPauseBacklogAttemptDecision::Tracked(_)));
prepare_scanner_pause_backlog_persist(&mut ledger, now + 2).expect("fixture unfinished attempt generation");
let published = persist_scanner_pause_backlog(Arc::clone(store), &loaded, ledger.clone())
.await
.expect("publish the complete native expansion fixture");
assert!(published.durable && published.stable_matches_ledger && !published.requires_reload);
assert!(ledger.has_unfinished_attempt());
assert_current_native_writer_ledger(store, &ledger).await;
set_runtime_error(None);
ledger
}
#[test]
#[serial_test::serial]
fn native_expansion_fixture_retains_controller_state_and_complete_disk_proof() {
run_native_retirement_test(async || {
use crate::storage_api::owner::NativeScannerPauseBacklogWriteFault;
let (_reference_root, reference) = native_pause_backlog_store(1, 2).await;
let now = unix_now();
let mut controller = ScannerPauseBacklogController::claim(Arc::clone(&reference), now)
.await
.expect("reference native writer claim");
controller.observe(observation(now + 1, true, 4)).await;
controller.observe(observation(now + 2, false, 4)).await;
assert!(matches!(
controller.begin_attempt(now + 2).await,
ScannerPauseBacklogAttemptDecision::Tracked(_)
));
let expected = controller.loaded.ledger.clone();
assert_current_native_writer_ledger(&reference, &expected).await;
let (_fixture_root, fixture) = native_pause_backlog_store(1, 2).await;
// A third preparation write would fail: the complete native commit
// and stabilization must suffice, without replaying the unrelated
// pause/resume transitions to disk in every recovery matrix cell.
let fault =
NativeScannerPauseBacklogWriteFault::fail_before_write(Arc::clone(&fixture.pools[0].disk_set[0]), "publish", 3);
let actual = seed_native_expansion_ledger(&fixture, now).await;
assert_eq!(actual, expected, "all generations, timestamps and unfinished work must be retained");
assert_current_native_writer_ledger(&fixture, &expected).await;
drop(fault);
drop(controller);
shutdown_native_retirement_store(reference).await;
shutdown_native_retirement_store(fixture).await;
});
}
#[test]
#[should_panic(expected = "original native scenario failure")]
fn native_retirement_runner_preserves_scenario_failure_payload() {
run_native_retirement_test_with_budget(Duration::from_secs(1), async || {
panic!("original native scenario failure");
});
}
async fn native_expanded_retirement_store(
old_pool_count: usize,
unstable_source: bool,
crates/scanner/src/scanner_folder.rs
+141 -34
View File
@@ -21,8 +21,8 @@ use std::time::{Duration, Instant, SystemTime};
use crate::ReplTargetSizeSummary;
use crate::data_usage_define::{
DATA_USAGE_SCAN_CHECKPOINT_VERSION, DataUsageCache, DataUsageCacheInfo, DataUsageEntry, DataUsageHash, DataUsageHashMap,
DataUsageRawEnumerationCursor, DataUsageScanCheckpoint, DataUsageScanCheckpointReason, PendingScannerHeal,
PendingScannerHealKind, ScannerSizeSummaryExt, SizeReconciliationEntry, SizeSummary, hash_path,
DataUsageRawEnumerationCursor, DataUsageScanCheckpoint, DataUsageScanCheckpointReason, MAX_DATA_USAGE_CACHE_DEPTH,
PendingScannerHeal, PendingScannerHealKind, ScannerSizeSummaryExt, SizeReconciliationEntry, SizeSummary, hash_path,
};
use crate::error::ScannerError;
use crate::raw_page_index::{RawEnumerationPageIndex, RawEnumerationPageWriter};
@@ -426,6 +426,10 @@ fn folder_resume_match(folder_name: &str, resume_after: &str) -> Option<FolderRe
.map(|_| FolderResumeMatch::Descendant)
}
fn folder_is_covered_by_frontier(folder: &str, frontier: &str) -> bool {
folder <= frontier && !matches!(folder_resume_match(folder, frontier), Some(FolderResumeMatch::Descendant))
}
fn order_items_for_resume<T, F>(items: &mut [T], resume_after: Option<&str>, name: F) -> FolderResumeOrder
where
F: Fn(&T) -> &str,
@@ -704,6 +708,12 @@ impl ScannerBucketPrefixScanScope {
}
/// Folder scanner for scanning directory structures
#[derive(Default)]
struct ScannerCheckpointAncestor {
hash: DataUsageHash,
entry: DataUsageEntry,
}
pub struct FolderScanner {
root: String,
old_cache: DataUsageCache,
@@ -728,6 +738,8 @@ pub struct FolderScanner {
updates: Option<mpsc::Sender<DataUsageEntry>>,
checkpoint_tx: Option<mpsc::Sender<DataUsageCache>>,
checkpoint_ancestors: Vec<ScannerCheckpointAncestor>,
checkpoint_depth: usize,
last_update: SystemTime,
checkpoint_objects: u64,
last_checkpoint_objects: u64,
@@ -1210,10 +1222,42 @@ impl FolderScanner {
.map_or((None, None), |progress| (progress.cursor(), progress.page_index()))
}
fn maybe_send_checkpoint(&mut self) {
async fn scan_child_folder(
&mut self,
ctx: CancellationToken,
folder: CachedFolder,
parent_hash: &mut DataUsageHash,
parent_entry: &mut DataUsageEntry,
child_entry: &mut DataUsageEntry,
) -> Result<(), ScannerError> {
if self.checkpoint_tx.is_none() || self.old_cache.info.scan_progress.is_none() {
return Box::pin(self.scan_folder(ctx, folder, child_entry)).await;
}
let depth = self.checkpoint_depth;
if depth >= MAX_DATA_USAGE_CACHE_DEPTH {
return Err(ScannerError::Other("scanner checkpoint ancestor depth limit exceeded".to_string()));
}
if self.checkpoint_ancestors.len() == depth {
self.checkpoint_ancestors.push(ScannerCheckpointAncestor::default());
}
// Reuse the empty frame slots: moving suspended accumulators avoids
// cloning a growing child set or allocating histogram vectors per child.
std::mem::swap(parent_hash, &mut self.checkpoint_ancestors[depth].hash);
std::mem::swap(parent_entry, &mut self.checkpoint_ancestors[depth].entry);
self.checkpoint_depth += 1;
let result = Box::pin(self.scan_folder(ctx, folder, child_entry)).await;
self.checkpoint_depth -= 1;
// Restore the parent before cancellation/error handling stitches partial work.
std::mem::swap(parent_hash, &mut self.checkpoint_ancestors[depth].hash);
std::mem::swap(parent_entry, &mut self.checkpoint_ancestors[depth].entry);
result
}
fn maybe_send_checkpoint(&mut self, folder: &CachedFolder, hash: &DataUsageHash, into: &DataUsageEntry) {
let Some(checkpoint_tx) = self.checkpoint_tx.clone() else { return };
let elapsed = self.last_checkpoint_at.elapsed();
if self.new_cache.info.scan_progress.is_none()
if into.compacted
|| self.new_cache.info.scan_progress.is_none()
|| elapsed < SCANNER_CHECKPOINT_MIN_INTERVAL
|| (self.checkpoint_objects.saturating_sub(self.last_checkpoint_objects) < SCANNER_CHECKPOINT_OBJECT_INTERVAL
&& elapsed < SCANNER_CHECKPOINT_INTERVAL)
@@ -1233,34 +1277,51 @@ impl FolderScanner {
}
let Ok(permit) = checkpoint_tx.try_reserve() else { return };
// Invalid partial graphs must not trigger a fresh O(N) clone on every object.
self.last_checkpoint_objects = self.checkpoint_objects;
self.last_checkpoint_at = Instant::now();
// Heal revisits are not sorted with the forward sweep. An unfinished
// earlier subtree, including a suspended ancestor, must not be signed
// as completed prefix coverage.
if self.coverage_frontier.as_deref().is_some_and(|frontier| {
folder_is_covered_by_frontier(&folder.name, frontier)
|| self.checkpoint_ancestors[..self.checkpoint_depth]
.iter()
.any(|ancestor| folder_is_covered_by_frontier(&ancestor.hash.0, frontier))
}) {
return;
}
let mut snapshot = self.new_cache.clone();
let mut parent = None;
for ancestor in &self.checkpoint_ancestors[..self.checkpoint_depth] {
snapshot.replace_hashed(&ancestor.hash, &parent, &ancestor.entry);
parent = Some(ancestor.hash.clone());
}
snapshot.replace_hashed(hash, &folder.parent, into);
if snapshot.checked_flatten_complete_scope(&snapshot.info.name).is_none() {
return;
}
snapshot.info.last_update = Some(SystemTime::now());
snapshot.info.snapshot_complete = false;
let (cursor, page_index) = self.peek_raw_enumeration_resume_state();
if cursor.is_some() || page_index.is_some() {
snapshot.info.scan_raw_enumeration_cursor = cursor;
snapshot.info.scan_raw_enumeration_page_index = page_index;
snapshot.info.scan_resume_after = None;
snapshot.info.scan_checkpoint = None;
snapshot.info.scan_coverage_receipt = None;
} else if snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).is_err() {
snapshot.info.scan_raw_enumeration_cursor = cursor;
snapshot.info.scan_raw_enumeration_page_index = page_index;
if snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).is_err() {
return;
}
if snapshot.root().is_none()
&& snapshot.info.scan_raw_enumeration_cursor.is_none()
&& snapshot.info.scan_raw_enumeration_page_index.is_none()
{
// A resumed readdir can run before covered subtrees are copied. Keep
// its durable receipt until this snapshot can represent that coverage.
if self.coverage_frontier.is_some() && snapshot.info.scan_coverage_receipt.is_none() {
return;
}
// The receipt was just sealed from this snapshot; do not hash it twice.
if snapshot.info.scan_raw_enumeration_cursor.is_none()
&& snapshot.info.scan_raw_enumeration_page_index.is_none()
&& snapshot.validated_scan_frontier().is_none()
&& snapshot.info.scan_coverage_receipt.is_none()
{
return;
}
permit.send(snapshot);
self.last_checkpoint_objects = self.checkpoint_objects;
self.last_checkpoint_at = Instant::now();
}
fn carry_forward_old_children(&mut self, parent_hash: &DataUsageHash, entry: &mut DataUsageEntry) {
@@ -1297,6 +1358,15 @@ impl FolderScanner {
child_entry: &DataUsageEntry,
) {
if data_usage_entry_has_progress(child_entry) {
// A cancelled out-of-order heal revisit is not completed prefix work.
if self
.coverage_frontier
.as_deref()
.is_some_and(|frontier| folder_is_covered_by_frontier(&child_hash.0, frontier))
{
self.coverage_frontier = None;
self.coverage_gap = true;
}
let mut child_entry = child_entry.clone();
self.carry_forward_old_children(child_hash, &mut child_entry);
self.record_scan_resume_hint_if_not_ancestor(&child_hash.key());
@@ -1513,7 +1583,7 @@ impl FolderScanner {
return Err(ScannerError::Other("Operation cancelled".to_string()));
}
let this_hash = hash_path(&folder.name);
let mut this_hash = hash_path(&folder.name);
// Store initial compaction state.
let was_compacted = into.compacted;
@@ -1526,7 +1596,7 @@ impl FolderScanner {
let mut abandoned_children: DataUsageHashMap = HashSet::new();
if !into.compacted {
abandoned_children = self.old_cache.find_children_copy(this_hash.clone());
abandoned_children = self.old_cache.find_children_copy(&this_hash);
}
debug!(
@@ -1587,8 +1657,26 @@ impl FolderScanner {
"Scanner folder state updated"
);
// A child prefix can be removed after its parent listed it and before
// it is opened here (e.g. concurrent object deletes). Treat it like a
// directory that vanished during iteration instead of failing the
// whole parent subtree. The bucket root (no parent) still propagates
// NotFound so namespace removal is detected by the caller.
let mut dir_reader = match tokio::fs::read_dir(&dir_path).await {
Ok(dir_reader) => dir_reader,
Ok(dir_reader) => Some(dir_reader),
Err(e) if e.kind() == ErrorKind::NotFound && folder.parent.is_some() => {
debug!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
dir_path = %dir_path,
state = "dir_missing_before_open",
error = %e,
"Scanner folder state updated"
);
None
}
Err(e) => return Err(ScannerError::Io(e)),
};
let mut pending_entry_progress = 0_u64;
@@ -1597,7 +1685,11 @@ impl FolderScanner {
let raw_enumeration_committed_entries = self.raw_enumeration_committed_entry_oracle(&folder.name);
loop {
let entry = match dir_reader.next_entry().await {
let Some(reader) = dir_reader.as_mut() else {
raw_enumeration_complete = true;
break;
};
let entry = match reader.next_entry().await {
Ok(Some(entry)) => entry,
Ok(None) => {
raw_enumeration_complete = true;
@@ -1651,7 +1743,7 @@ impl FolderScanner {
}
}
self.record_raw_enumeration_entry(&folder.name, &file_name);
self.maybe_send_checkpoint();
self.maybe_send_checkpoint(&folder, &this_hash, into);
let is_storage_format_entry = file_name == STORAGE_FORMAT_FILE;
let file_path = entry.path().to_string_lossy().to_string();
@@ -1958,7 +2050,7 @@ impl FolderScanner {
object_count += 1;
self.budget.record_object_scanned();
self.checkpoint_objects = self.checkpoint_objects.saturating_add(1);
self.maybe_send_checkpoint();
self.maybe_send_checkpoint(&folder, &this_hash, into);
timer.sleep().await;
@@ -2159,10 +2251,9 @@ impl FolderScanner {
let h = hash_path(&folder_item.name);
if forward_sweep
&& !into.compacted
&& forward_resume_after.as_deref().is_some_and(|resume| {
folder_item.name.as_str() <= resume
&& !matches!(folder_resume_match(&folder_item.name, resume), Some(FolderResumeMatch::Descendant))
})
&& forward_resume_after
.as_deref()
.is_some_and(|resume| folder_is_covered_by_frontier(&folder_item.name, resume))
&& self.old_cache.find(&folder_item.name).is_some()
{
self.new_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
@@ -2233,8 +2324,10 @@ impl FolderScanner {
let mut dst = DataUsageEntry::default();
// Use Box::pin for recursive async call
let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), &mut dst));
if let Err(e) = fut.await {
if let Err(e) = self
.scan_child_folder(ctx.clone(), folder_item.clone(), &mut this_hash, into, &mut dst)
.await
{
if ctx.is_cancelled() {
self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst)
.await;
@@ -2258,7 +2351,7 @@ impl FolderScanner {
into.add_child(&h);
self.record_completed_child(&folder_item.name, dst.failed_objects == 0);
self.maybe_send_checkpoint();
self.maybe_send_checkpoint(&folder, &this_hash, into);
// We scanned a folder, optionally send update.
self.update_cache.delete_recursive(&h);
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
@@ -2635,8 +2728,10 @@ impl FolderScanner {
let h = hash_path(&folder_item.name);
// Use Box::pin for recursive async call
let fut = Box::pin(self.scan_folder(ctx.clone(), folder_item.clone(), &mut dst));
if let Err(e) = fut.await {
if let Err(e) = self
.scan_child_folder(ctx.clone(), folder_item.clone(), &mut this_hash, into, &mut dst)
.await
{
if ctx.is_cancelled() {
self.preserve_partial_child_progress(&folder_item.parent, &h, into, &dst)
.await;
@@ -2662,7 +2757,7 @@ impl FolderScanner {
self.update_cache.delete_recursive(&h);
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
self.send_update().await;
self.maybe_send_checkpoint();
self.maybe_send_checkpoint(&folder, &this_hash, into);
}
}
}
@@ -2803,6 +2898,9 @@ pub(crate) async fn scan_data_folder_scoped(
cache.info.tier_registry_generation = Some(tier_registry.generation);
let resume_frontier = cache.validated_scan_frontier().map(str::to_owned);
if resume_frontier.is_some() {
global_metrics().record_scanner_checkpoint_used();
}
let bucket_name = cache.info.name.clone();
let next_cycle = cache.info.next_cycle;
let had_scan_checkpoint = cache.info.scan_checkpoint.is_some();
@@ -2834,6 +2932,8 @@ pub(crate) async fn scan_data_folder_scoped(
disks_quorum,
updates,
checkpoint_tx,
checkpoint_ancestors: Vec::new(),
checkpoint_depth: 0,
last_update: SystemTime::UNIX_EPOCH,
checkpoint_objects: 0,
last_checkpoint_objects: 0,
@@ -2988,6 +3088,13 @@ pub(crate) async fn scan_data_folder_scoped(
}
if matches!(&e, ScannerError::Io(io) if io.kind() == ErrorKind::NotFound) {
let mut partial_cache = scanner.old_cache;
// An empty input still needs a root for the non-publishable
// partial result; retain nonempty durable caches unchanged.
if partial_cache.cache.is_empty() {
partial_cache
.cache
.insert(hash_path(&bucket_name).0, DataUsageEntry::default());
}
partial_cache.info.last_update = Some(SystemTime::now());
partial_cache.info.next_cycle = next_cycle;
partial_cache.info.snapshot_complete = false;
crates/scanner/src/scanner_folder/tests.rs
+227 -2
View File
@@ -343,6 +343,8 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
disks_quorum: 0,
updates: None,
checkpoint_tx: None,
checkpoint_ancestors: Vec::new(),
checkpoint_depth: 0,
last_update: SystemTime::UNIX_EPOCH,
checkpoint_objects: 0,
last_checkpoint_objects: 0,
@@ -409,14 +411,21 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait()
.seal_scan_frontier(Some("bucket/a"))
.expect("fixture frontier");
assert_eq!(scanner.new_cache.validated_scan_frontier(), Some("bucket/a"));
let checkpoint_folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
let checkpoint_hash = hash_path("bucket");
let checkpoint_entry = scanner.new_cache.root().expect("fixture root");
scanner.maybe_send_checkpoint();
scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL * 2;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("test instant subtraction");
scanner.maybe_send_checkpoint();
scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry);
assert_eq!(
scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL,
"a full checkpoint queue must reject before cloning or advancing progress"
@@ -427,11 +436,73 @@ async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait()
assert!(!checkpoint.info.snapshot_complete);
assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL);
assert!(checkpoint_rx.try_recv().is_err(), "checkpoint queue remains bounded");
scanner
.new_cache
.cache
.insert("bucket/orphan".to_string(), DataUsageEntry::default());
scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry);
assert!(
checkpoint_rx.try_recv().is_err(),
"unreachable observations must not become resumable snapshots"
);
assert_eq!(
scanner.last_checkpoint_objects,
SCANNER_CHECKPOINT_OBJECT_INTERVAL * 2,
"an invalid attempt must still consume the producer interval"
);
scanner.new_cache.cache.remove("bucket/orphan");
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL * 3;
scanner.maybe_send_checkpoint(&checkpoint_folder, &checkpoint_hash, &checkpoint_entry);
assert!(checkpoint_rx.try_recv().is_err(), "a repaired graph still waits for the minimum interval");
tokio::fs::remove_dir_all(temp_dir)
.await
.expect("remove test scanner directory");
}
#[tokio::test]
async fn checkpoint_ancestor_depth_limit_preserves_suspended_accounting() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(temp_dir),
};
let (tx, _rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(tx);
scanner.old_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
});
scanner.checkpoint_depth = MAX_DATA_USAGE_CACHE_DEPTH;
scanner.checkpoint_ancestors = (0..MAX_DATA_USAGE_CACHE_DEPTH)
.map(|_| ScannerCheckpointAncestor::default())
.collect();
let mut hash = hash_path("bucket/parent");
let mut parent = DataUsageEntry {
objects: 3,
size: 7,
..Default::default()
};
let mut child = DataUsageEntry::default();
let error = scanner
.scan_child_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket/parent/child".to_string(),
parent: Some(hash.clone()),
object_heal_prob_div: 1,
},
&mut hash,
&mut parent,
&mut child,
)
.await
.expect_err("unsupported depth must stop before recursion");
assert!(matches!(error, ScannerError::Other(message) if message.contains("depth limit")));
assert_eq!(hash, hash_path("bucket/parent"));
assert_eq!((parent.objects, parent.size), (3, 7));
assert_eq!(scanner.checkpoint_depth, MAX_DATA_USAGE_CACHE_DEPTH);
}
struct TestGuard {
temp_dir: Option<std::path::PathBuf>,
}
@@ -1658,6 +1729,51 @@ fn metadata_for_object_version(bucket: &str, object: &str, version_id: Option<Uu
meta.marshal_msg().expect("test metadata should marshal")
}
#[tokio::test]
#[serial]
async fn test_scan_folder_keeps_old_cache_bounded_by_loaded_entries() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
for directory in 0..16 {
for object in 0..32 {
write_test_object_metadata(&temp_dir, "bucket", &format!("dir-{directory:03}/obj-{object:03}")).await;
}
}
let initial_capacity = scanner.old_cache.cache.capacity();
let mut into = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut into,
)
.await
.expect("scan object directory tree");
assert_eq!(
scanner
.new_cache
.size_recursive(&hash_path("bucket").0)
.expect("scanned root totals")
.objects,
512,
"all object directories must still be counted",
);
assert!(scanner.old_cache.cache.is_empty(), "lookups must not retain visited object placeholders");
assert_eq!(
scanner.old_cache.cache.capacity(),
initial_capacity,
"old cache must not allocate a growing table"
);
}
async fn write_test_object_metadata(root: &std::path::Path, bucket: &str, object: &str) {
write_test_object_metadata_bytes(root, bucket, object, &metadata_for_object(bucket, object)).await;
}
@@ -2056,6 +2172,59 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
assert!(result.is_ok(), "expected unreadable child directory to be skipped");
}
#[tokio::test]
#[serial]
async fn test_scan_folder_treats_child_removed_before_open_as_empty() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());
// The parent listed `gone/`, then a concurrent delete removed it before the
// child scan opened it.
std::fs::create_dir_all(temp_dir.join("bucket")).expect("failed to create bucket dir");
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket/gone".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
};
let mut into = DataUsageEntry::default();
let result = scanner.scan_folder(CancellationToken::new(), folder, &mut into).await;
assert!(result.is_ok(), "expected a child removed before open to be skipped, got {result:?}");
assert_eq!(into.objects, 0);
assert_eq!(into.size, 0);
}
#[tokio::test]
#[serial]
async fn test_scan_folder_missing_bucket_root_still_reports_not_found() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir.clone());
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
let mut into = DataUsageEntry::default();
let result = scanner.scan_folder(CancellationToken::new(), folder, &mut into).await;
assert!(
matches!(&result, Err(ScannerError::Io(e)) if e.kind() == ErrorKind::NotFound),
"expected a missing bucket root to keep reporting NotFound, got {result:?}"
);
}
#[tokio::test]
#[serial]
async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
@@ -3277,6 +3446,62 @@ async fn scan_data_folder_missing_scan_root_returns_partial() {
assert_eq!(root.size, 0);
}
#[tokio::test]
#[serial]
async fn scan_data_folder_missing_scan_root_preserves_disconnected_durable_cache() {
let (scanner, temp_dir) = build_test_scanner().await;
tokio::fs::remove_dir_all(&temp_dir).await.expect("remove scanner root");
let _guard = TestGuard {
temp_dir: Some(temp_dir),
};
let parent = CancellationToken::new();
let budget = ScannerCycleBudget::new(&parent, Default::default());
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: "missing-bucket".to_string(),
next_cycle: 9,
snapshot_complete: true,
..Default::default()
},
..Default::default()
};
let orphan = hash_path("missing-bucket/orphan");
cache.cache.insert(
orphan.0.clone(),
DataUsageEntry {
objects: 7,
size: 70,
..Default::default()
},
);
let retained_key_ptr = cache.cache.keys().next().expect("durable orphan key").as_ptr();
let result = scan_data_folder(
budget.token(),
budget,
vec![scanner.local_disk.clone()],
scanner.local_disk,
cache,
None,
HealScanMode::Normal,
SCANNER_SLEEPER.clone(),
)
.await;
let partial = match result {
Err(ScannerError::NamespaceNotFoundCache(partial)) => partial,
other => panic!("missing namespace must remain incomplete, got {other:?}"),
};
assert!(!partial.info.snapshot_complete);
assert_eq!(partial.info.next_cycle, 9);
assert_eq!(partial.cache.len(), 1);
assert_eq!(partial.cache.keys().next().expect("retained orphan key").as_ptr(), retained_key_ptr);
assert!(
partial.root().is_none(),
"a disconnected durable cache must not gain an invented zero root"
);
assert!(partial.checked_flatten_complete_scope("missing-bucket").is_none());
assert_eq!((partial.cache[&orphan.0].objects, partial.cache[&orphan.0].size), (7, 70));
}
#[tokio::test]
#[serial]
async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() {
crates/scanner/src/scanner_folder/tests/checkpoint_fixture.rs
+1
View File
@@ -21,6 +21,7 @@ use std::io::Cursor;
use tokio::io::AsyncReadExt;
mod deep_compacted;
mod periodic_snapshot;
mod segment_observation;
const CACHE_NAME: &str = "bucket/checkpoint-fixture.bin";
crates/scanner/src/scanner_folder/tests/checkpoint_fixture/periodic_snapshot.rs
+794
View File
@@ -0,0 +1,794 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Periodic snapshots captured while nested traversal frames are still active.
use super::*;
use tokio::sync::Notify;
const OBJECTS: u64 = 8;
fn prepare_scanner(scanner: &mut FolderScanner, cache: DataUsageCache) {
scanner.resume_frontier = cache.validated_scan_frontier().map(str::to_owned);
scanner.coverage_frontier = scanner.resume_frontier.clone();
scanner.new_cache = DataUsageCache {
info: cache.info.clone(),
..Default::default()
};
scanner.update_cache = DataUsageCache {
info: cache.info.clone(),
..Default::default()
};
scanner.old_cache = cache;
scanner.is_erasure_mode = true;
scanner.skip_heal.store(true, Ordering::SeqCst);
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest);
}
async fn nested_periodic_snapshot(
corrupt_first: bool,
root_scalars: (usize, usize),
) -> (DataUsageCache, crate::DataUsageScanIdentity, FolderScanner, TestGuard) {
let (mut scanner, root) = build_test_scanner().await;
let guard = TestGuard {
temp_dir: Some(root.clone()),
};
for branch in ["a", "b"] {
for object in 0..4 {
write_checkpoint_object(&root, &format!("prefix/{branch}/{object:04}"), &[(None, 1)]).await;
}
}
if corrupt_first {
write_test_object_metadata_bytes(&root, "bucket", "prefix/a/0000", b"not-valid-filemeta").await;
}
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
prepare_scanner(&mut scanner, cache);
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("seed elapsed checkpoint interval");
let paused = Arc::new(Notify::new());
let release = Arc::new(Notify::new());
let pause_path = if corrupt_first {
"bucket/prefix/a/0003"
} else {
"bucket/prefix/a/0002"
};
scanner.update_current_path = Arc::new({
let paused = paused.clone();
let release = release.clone();
move |path: &str| {
let pause_here = path == pause_path;
let paused = paused.clone();
let release = release.clone();
Box::pin(async move {
if pause_here {
paused.notify_one();
release.notified().await;
}
})
}
});
let parent = CancellationToken::new();
scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default());
let budget = scanner.budget.clone();
let mut entry = DataUsageEntry {
objects: root_scalars.0,
size: root_scalars.1,
..Default::default()
};
let snapshot = {
let scan = scanner.scan_folder(
parent.clone(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut entry,
);
tokio::pin!(scan);
let snapshot = tokio::time::timeout(Duration::from_secs(10), async {
tokio::select! {
snapshot = checkpoint_rx.recv() => snapshot.expect("real walk emits periodic snapshot"),
result = &mut scan => panic!("walk finished before periodic snapshot: {result:?}"),
}
})
.await
.expect("bounded nested checkpoint emission");
tokio::time::timeout(Duration::from_secs(10), async {
tokio::select! {
() = paused.notified() => {},
result = &mut scan => panic!("nested ancestors unwound before capture: {result:?}"),
}
})
.await
.expect("walk pauses before third healthy object");
assert_eq!(budget.progress().0, 2, "capture must precede ancestor unwind and further object reads");
assert!(!parent.is_cancelled(), "periodic capture must not use cancellation linking");
snapshot
};
assert!(!snapshot.info.snapshot_complete);
assert!(
snapshot.validated_raw_enumeration_cursor().is_some() || snapshot.validated_raw_enumeration_page_index().is_some(),
"enumeration progress must coexist with completed subtree coverage"
);
let totals = snapshot
.checked_flatten_complete_scope("bucket")
.expect("periodic snapshot must connect every active ancestor to the bucket root");
assert_eq!(
totals.objects,
2 + root_scalars.0,
"snapshot must retain suspended ancestor and current frame object accounting"
);
assert_eq!(
totals.size,
2 + root_scalars.1,
"suspended ancestor and current frame scalar bytes must survive snapshot construction"
);
let store = FixtureStore::new();
let revisions = DataUsageCache::default()
.load_with_revisions(store.clone(), CACHE_NAME)
.await
.expect("initial periodic fixture revisions");
snapshot
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
.await
.expect("persist real mid-walk periodic snapshot");
let mut loaded = DataUsageCache::default();
loaded
.load(store.clone(), CACHE_NAME)
.await
.expect("reload real periodic snapshot");
assert_eq!(loaded.info.scan_coverage_receipt, snapshot.info.scan_coverage_receipt);
assert_eq!(loaded.info.scan_raw_enumeration_cursor, snapshot.info.scan_raw_enumeration_cursor);
assert_eq!(
loaded
.checked_flatten_complete_scope("bucket")
.expect("reloaded complete scope")
.objects,
2 + root_scalars.0
);
assert_eq!(
loaded.prepare_bucket_checkpoint("bucket", 11, 8, SOURCE, PLAN, identity),
crate::DataUsageCachePrepareOutcome::Reused,
"a higher leader must adopt the persisted periodic snapshot"
);
(loaded, identity, scanner, guard)
}
async fn finish_resumed_walk(cache: DataUsageCache, original: &FolderScanner) -> (DataUsageCache, u64, TestGuard) {
let (mut scanner, unused_root) = build_test_scanner().await;
let guard = TestGuard {
temp_dir: Some(unused_root),
};
scanner.root = original.root.clone();
scanner.local_disk = original.local_disk.clone();
prepare_scanner(&mut scanner, cache);
let parent = CancellationToken::new();
scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default());
let mut entry = DataUsageEntry::default();
scanner
.scan_folder(
parent,
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut entry,
)
.await
.expect("finish resumed nested periodic walk");
let visits = scanner.budget.progress().0;
(scanner.new_cache, visits, guard)
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_resumes_after_leader_handoff_with_exact_totals() {
let (loaded, _, original, _guard) = nested_periodic_snapshot(false, (0, 0)).await;
assert_eq!(loaded.validated_scan_frontier(), Some("bucket/prefix/a/0000"));
let (finished, visits, _unused_guard) = finish_resumed_walk(loaded, &original).await;
assert_eq!(visits, OBJECTS - 1, "the completed nested sibling must skip its metadata read");
let totals = finished
.checked_flatten_complete_scope("bucket")
.expect("finished connected root");
assert_eq!(u64::try_from(totals.objects).expect("object count"), OBJECTS);
assert_eq!(u64::try_from(totals.size).expect("byte count"), OBJECTS);
assert_eq!(totals.failed_objects, 0);
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_keeps_failed_prefix_outside_resume_frontier() {
let (loaded, _, original, _guard) = nested_periodic_snapshot(true, (0, 0)).await;
assert!(
loaded.validated_scan_frontier().is_none(),
"an earlier corrupt object must block subtree skipping"
);
assert!(!loaded.info.failed_objects.is_empty(), "persisted failure debt must survive handoff");
let (finished, visits, _unused_guard) = finish_resumed_walk(loaded, &original).await;
assert_eq!(visits, OBJECTS - 1, "every healthy object must be re-read after the coverage gap");
let totals = finished
.checked_flatten_complete_scope("bucket")
.expect("failed sweep still has connected accounting");
assert_eq!(u64::try_from(totals.objects).expect("healthy count"), OBJECTS - 1);
assert_eq!(u64::try_from(totals.size).expect("healthy bytes"), OBJECTS - 1);
assert!(!finished.info.failed_objects.is_empty(), "unrepaired metadata must remain pending");
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_preserves_suspended_ancestor_scalars_through_reload() {
let (loaded, _, _, _guard) = nested_periodic_snapshot(false, (3, 21)).await;
let root = loaded.root().expect("reloaded root accumulator");
assert_eq!(root.objects, 3, "suspended root object accounting must not become an empty topology node");
assert_eq!(root.size, 21, "suspended root bytes must remain independent of descendant totals");
let totals = loaded
.checked_flatten_complete_scope("bucket")
.expect("reloaded ancestor accounting");
assert_eq!(totals.objects, 5);
assert_eq!(totals.size, 23);
assert_eq!(loaded.validated_scan_frontier(), Some("bucket/prefix/a/0000"));
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_keeps_durable_frontier_across_early_repeated_handoffs() {
let (cache, identity, original, _guard) = nested_periodic_snapshot(false, (0, 0)).await;
let store = FixtureStore::new();
let revisions = DataUsageCache::default()
.load_with_revisions(store.clone(), CACHE_NAME)
.await
.expect("initial repeated-handoff revisions");
cache
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
.await
.expect("save the inherited valid subtree frontier");
let (mut scanner, unused_root) = build_test_scanner().await;
let _unused_guard = TestGuard {
temp_dir: Some(unused_root),
};
scanner.root = original.root.clone();
scanner.local_disk = original.local_disk.clone();
prepare_scanner(&mut scanner, cache);
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("early resumed emitter cadence");
let folder = CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
};
scanner.record_raw_enumeration_entry("bucket", "prefix");
scanner.maybe_send_checkpoint(&folder, &hash_path("bucket"), &DataUsageEntry::default());
assert!(
matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"an early raw-only snapshot must not overwrite inherited completed coverage"
);
assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL);
let mut inherited = DataUsageCache::default();
let revisions = inherited
.load_with_revisions(store.clone(), CACHE_NAME)
.await
.expect("reload unchanged durable frontier after deferred early emission");
assert_eq!(inherited.validated_scan_frontier(), Some("bucket/prefix/a/0000"));
assert_eq!(
inherited.prepare_bucket_checkpoint("bucket", 11, 9, SOURCE, PLAN, identity),
crate::DataUsageCachePrepareOutcome::Reused
);
assert_eq!(inherited.validated_scan_frontier(), Some("bucket/prefix/a/0000"));
prepare_scanner(&mut scanner, inherited);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2;
scanner.last_checkpoint_objects = 0;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("later useful resumed emitter cadence");
let parent = CancellationToken::new();
scanner.budget = ScannerCycleBudget::new_with_progress_tracking(&parent, Default::default());
let mut entry = DataUsageEntry::default();
let candidate = {
let scan = scanner.scan_folder(parent, folder, &mut entry);
tokio::pin!(scan);
tokio::time::timeout(Duration::from_secs(10), async {
tokio::select! {
checkpoint = checkpoint_rx.recv() => checkpoint.expect("resumed walk can later emit useful coverage"),
result = &mut scan => {
result.expect("later resumed walk completes");
checkpoint_rx.try_recv().expect("completed walk emitted a useful periodic checkpoint")
}
}
})
.await
.expect("bounded later useful emission")
};
assert_eq!(candidate.validated_scan_frontier(), Some("bucket/prefix/a/0001"));
assert_eq!(
candidate
.checked_flatten_complete_scope("bucket")
.expect("later snapshot connected accounting")
.objects,
3,
"inherited completed object and both newly read objects must survive"
);
candidate
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
.await
.expect("persist later coverage without losing the inherited prefix");
let mut latest = DataUsageCache::default();
latest
.load(store, CACHE_NAME)
.await
.expect("reload later periodic checkpoint");
assert_eq!(
latest.prepare_bucket_checkpoint("bucket", 11, 10, SOURCE, PLAN, identity),
crate::DataUsageCachePrepareOutcome::Reused
);
assert_eq!(latest.validated_scan_frontier(), Some("bucket/prefix/a/0001"));
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_defers_unfinished_paths_at_or_before_completed_frontier() {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard { temp_dir: Some(root) };
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
cache.replace("bucket", "", DataUsageEntry::default());
cache.replace(
"bucket/z",
"bucket",
DataUsageEntry {
objects: 3,
size: 9,
..Default::default()
},
);
cache
.seal_scan_frontier(Some("bucket/z"))
.expect("completed later sibling coverage");
prepare_scanner(&mut scanner, cache.clone());
scanner.new_cache = cache;
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
let partial = DataUsageEntry {
objects: 2,
size: 5,
..Default::default()
};
for active in ["bucket/a", "bucket/z"] {
scanner.record_raw_enumeration_entry(active, STORAGE_FORMAT_FILE);
scanner.checkpoint_objects += SCANNER_CHECKPOINT_OBJECT_INTERVAL;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("unordered active path cadence");
scanner.maybe_send_checkpoint(
&CachedFolder {
name: active.to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
},
&hash_path(active),
&partial,
);
assert!(
matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"unfinished path {active} must not become certified completed coverage"
);
assert_eq!(
scanner.last_checkpoint_objects, scanner.checkpoint_objects,
"deferred paths still consume cadence"
);
}
scanner.new_cache.replace("bucket/a", "bucket", partial);
let root_entry = scanner.new_cache.root().expect("completed subtree owner");
scanner.checkpoint_objects += SCANNER_CHECKPOINT_OBJECT_INTERVAL;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("completed owner cadence");
scanner.maybe_send_checkpoint(
&CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&hash_path("bucket"),
&root_entry,
);
let checkpoint = checkpoint_rx.try_recv().expect("completed subtree owner may safely emit");
assert_eq!(checkpoint.validated_scan_frontier(), Some("bucket/z"));
let totals = checkpoint
.checked_flatten_complete_scope("bucket")
.expect("both completed sibling links remain reachable");
assert_eq!(totals.objects, 5);
assert_eq!(totals.size, 14);
assert_eq!(checkpoint.cache.len(), 3);
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_checks_suspended_ancestors_before_certifying_frontier() {
for (ancestor, frontier, may_emit) in [
("bucket/z", "bucket/z", false),
("bucket/a", "bucket/a-", false),
("bucket/a", "bucket/a/done", true),
] {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard { temp_dir: Some(root) };
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
cache.replace("bucket", "", DataUsageEntry::default());
cache.replace(ancestor, "bucket", DataUsageEntry::default());
let completed = format!("{ancestor}/done");
cache.replace(
&completed,
ancestor,
DataUsageEntry {
objects: 3,
size: 9,
..Default::default()
},
);
let sibling_frontier = frontier != ancestor && frontier != completed;
if sibling_frontier {
cache.replace(
frontier,
"bucket",
DataUsageEntry {
objects: 3,
size: 9,
..Default::default()
},
);
}
cache.seal_scan_frontier(Some(frontier)).expect("initial completed coverage");
prepare_scanner(&mut scanner, cache);
scanner.new_cache.replace("bucket", "", DataUsageEntry::default());
scanner
.new_cache
.copy_with_children(&scanner.old_cache, &hash_path(&completed), &Some(hash_path(ancestor)));
if sibling_frontier {
scanner
.new_cache
.copy_with_children(&scanner.old_cache, &hash_path(frontier), &Some(hash_path("bucket")));
}
let root_entry = scanner.new_cache.root().expect("suspended root owner").clone();
let mut ancestor_entry = scanner.new_cache.find(ancestor).expect("suspended subtree owner").clone();
ancestor_entry.objects = 1;
ancestor_entry.size = 2;
scanner.checkpoint_ancestors = vec![
ScannerCheckpointAncestor {
hash: hash_path("bucket"),
entry: root_entry,
},
ScannerCheckpointAncestor {
hash: hash_path(ancestor),
entry: ancestor_entry,
},
];
scanner.checkpoint_depth = 2;
let current = format!("{ancestor}/next");
assert!(
current.as_str() > frontier,
"the current-folder-only guard must not already reject this fixture"
);
scanner.record_raw_enumeration_entry(&current, STORAGE_FORMAT_FILE);
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("suspended ancestor emitter cadence");
scanner.maybe_send_checkpoint(
&CachedFolder {
name: current.clone(),
parent: Some(hash_path(ancestor)),
object_heal_prob_div: 1,
},
&hash_path(&current),
&DataUsageEntry {
objects: 2,
size: 5,
..Default::default()
},
);
assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL);
if may_emit {
let snapshot = checkpoint_rx
.try_recv()
.expect("an ancestor containing the frontier may safely emit");
assert_eq!(snapshot.validated_scan_frontier(), Some(frontier));
let totals = snapshot
.checked_flatten_complete_scope("bucket")
.expect("all suspended and current accounting connected");
assert_eq!(totals.objects, 6);
assert_eq!(totals.size, 16);
assert_eq!(snapshot.cache.len(), 4);
} else {
assert!(
matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"unfinished ancestor {ancestor} must not be certified through {frontier}"
);
}
}
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_partial_preservation_invalidates_covered_child_but_keeps_inner_frontier() {
for (child, frontier, frontier_inside_child, expected_objects) in [
("bucket/a", "bucket/z", false, 5),
("bucket/z", "bucket/z", false, 2),
("bucket/parent", "bucket/parent/done", true, 5),
] {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard { temp_dir: Some(root) };
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
cache.replace("bucket", "", DataUsageEntry::default());
if frontier_inside_child {
cache.replace(child, "bucket", DataUsageEntry::default());
}
cache.replace(
frontier,
if frontier_inside_child { child } else { "bucket" },
DataUsageEntry {
objects: 3,
size: 9,
..Default::default()
},
);
cache
.seal_scan_frontier(Some(frontier))
.expect("initial completed child coverage");
prepare_scanner(&mut scanner, cache.clone());
scanner.new_cache = cache;
let mut parent_entry = scanner.new_cache.root().expect("partial child owner");
let child_entry = DataUsageEntry {
objects: 2,
size: 5,
..Default::default()
};
scanner
.preserve_partial_child_progress(&Some(hash_path("bucket")), &hash_path(child), &mut parent_entry, &child_entry)
.await;
let retained_frontier = scanner.coverage_frontier.clone();
scanner.new_cache.replace_hashed(&hash_path("bucket"), &None, &parent_entry);
scanner
.new_cache
.seal_scan_frontier(retained_frontier.as_deref())
.expect("seal the final partial cache after child preservation");
let totals = scanner
.new_cache
.checked_flatten_complete_scope("bucket")
.expect("preserved partial child and completed siblings remain connected");
assert_eq!(totals.objects, expected_objects, "partial child {child} must be counted once");
assert_eq!(totals.size, if expected_objects == 2 { 5 } else { 14 });
if frontier_inside_child {
assert!(!scanner.coverage_gap);
assert_eq!(scanner.new_cache.validated_scan_frontier(), Some(frontier));
} else {
assert!(
scanner.coverage_gap,
"unfinished covered child {child} must block further frontier advancement"
);
assert!(retained_frontier.is_none());
assert!(scanner.new_cache.info.scan_coverage_receipt.is_none());
assert!(scanner.new_cache.validated_scan_frontier().is_none());
}
}
}
async fn assert_nested_frame_restoration(max_objects: Option<u64>) {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(root.clone()),
};
for branch in ["a", "b"] {
for object in 0..4 {
write_checkpoint_object(&root, &format!("prefix/{branch}/{object:04}"), &[(None, 1)]).await;
}
}
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
prepare_scanner(&mut scanner, cache);
let (checkpoint_tx, _checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
let parent = CancellationToken::new();
scanner.budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
ScannerCycleBudgetConfig {
max_objects,
..Default::default()
},
);
let mut entry = DataUsageEntry {
objects: usize::from(max_objects.is_some()) * 3,
size: usize::from(max_objects.is_some()) * 21,
..Default::default()
};
let result = scanner
.scan_folder(
scanner.budget.token(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut entry,
)
.await;
match max_objects {
Some(_) => {
assert!(result.is_err(), "object budget must interrupt the actual nested walk");
assert_eq!(scanner.budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Objects));
assert!(
scanner.budget.token().is_cancelled(),
"partial preservation must follow actual cancellation"
);
assert_eq!(
entry.objects, 3,
"suspended root object accounting must be restored before returning the error"
);
assert_eq!(
entry.size, 21,
"suspended root byte accounting must be restored before cancellation linking"
);
}
None => result.expect("complete real nested walk with periodic sender enabled"),
}
let totals = scanner
.new_cache
.checked_flatten_complete_scope("bucket")
.expect("complete nested accounting");
let expected = max_objects.unwrap_or(OBJECTS);
assert_eq!(u64::try_from(totals.objects).expect("object count"), expected);
assert_eq!(u64::try_from(totals.size).expect("byte count"), expected);
assert_eq!(scanner.checkpoint_depth, 0, "every completed child must restore its parent frame");
assert_eq!(
scanner.checkpoint_ancestors.len(),
3,
"sibling count must not increase depth-bounded spare storage"
);
for spare in &scanner.checkpoint_ancestors {
assert!(spare.hash.0.is_empty(), "reusable frame must release its suspended path");
assert!(
!data_usage_root_has_progress(&spare.entry),
"completed frame must not retain ancestor accounting"
);
assert!(!spare.entry.compacted);
assert!(spare.entry.obj_sizes.is_empty());
assert!(spare.entry.obj_versions.is_empty());
assert!(spare.entry.tier_accounting_proof.is_none());
}
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_reuses_empty_ancestor_frames_after_complete_walk() {
assert_nested_frame_restoration(None).await;
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_restores_ancestor_frames_before_cancellation_linking() {
assert_nested_frame_restoration(Some(2)).await;
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_adoption_records_checkpoint_used_on_real_scan_startup() {
let (scanner, root) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(root.clone()),
};
tokio::fs::create_dir_all(root.join("bucket/static"))
.await
.expect("create the real cached subtree directory");
let (mut cache, _) = bound_checkpoint();
cache.info.skip_healing = true;
let parent = CancellationToken::new();
let budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
ScannerCycleBudgetConfig {
max_directories: Some(1),
..Default::default()
},
);
let before = global_metrics().report().await.scan_checkpoint_used;
let completed = scan_data_folder(
budget.token(),
budget.clone(),
vec![scanner.local_disk.clone()],
scanner.local_disk,
cache,
None,
HealScanMode::Normal,
DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest),
)
.await
.expect("validated frontier must skip the child within a root-only directory budget");
let after = global_metrics().report().await.scan_checkpoint_used;
assert_eq!(after, before + 1, "adopting a verified forward-sweep frontier must be observable");
assert!(completed.info.snapshot_complete);
assert_eq!(
completed
.checked_flatten_complete_scope("bucket")
.expect("completed adopted scope")
.objects,
3
);
assert_eq!(budget.progress().1, 1, "adoption must avoid entering the completed subtree");
assert_eq!(budget.reason(), None);
}
#[tokio::test]
#[serial]
async fn periodic_nested_snapshot_does_not_publish_shared_compacted_accumulator_under_child_path() {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(root.clone()),
};
for object in 0..4 {
write_checkpoint_object(&root, &format!("prefix/nested/{object:04}"), &[(None, 1)]).await;
}
let (_, identity) = bound_checkpoint();
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
prepare_scanner(&mut scanner, cache);
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
scanner.checkpoint_tx = Some(checkpoint_tx);
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL - 2;
scanner.last_checkpoint_at = Instant::now()
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
.expect("seed compacted checkpoint interval");
let mut entry = DataUsageEntry {
compacted: true,
..Default::default()
};
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut entry,
)
.await
.expect("walk with shared compacted owner accumulator");
assert_eq!(entry.objects, 4);
assert_eq!(entry.size, 4);
assert!(
entry.children.is_empty(),
"compacted totals must not acquire independently accounted child links"
);
assert!(
matches!(checkpoint_rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)),
"a shared compacted parent accumulator must not be snapshotted as the current child"
);
}
crates/scanner/src/scanner_io/cache.rs
+45 -3
View File
@@ -28,7 +28,8 @@ pub(crate) fn scanner_cache_lock_timeout() -> Duration {
pub(crate) enum ScannerCheckpointPersistResult {
Saved,
FenceChanged,
Failed(StorageError),
Skipped,
RetryBucket(StorageError),
}
pub(crate) struct ScannerCheckpointPersistContext<'a> {
@@ -80,7 +81,12 @@ where
)
.await;
if let Err(error) = checkpoint_save {
return ScannerCheckpointPersistResult::Failed(error);
record_checkpoint_save_failure(cache_name, checkpoint, &error);
return if matches!(error, StorageError::PreconditionFailed) {
ScannerCheckpointPersistResult::RetryBucket(error)
} else {
ScannerCheckpointPersistResult::Skipped
};
}
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, fence_store)
@@ -101,12 +107,48 @@ where
{
Ok(next_revisions) => {
*revisions = next_revisions;
if let Some(source) = checkpoint.info.source {
let saved = checkpoint.info.scan_checkpoint.as_ref().map(|cursor| {
rustfs_scanner_metrics::metrics::ScannerCheckpointReport {
version: cursor.version,
resume_after: cursor.resume_after.clone(),
reason: cursor.reason.as_str().to_string(),
last_event: "saved".to_string(),
}
});
global_metrics().record_scanner_checkpoint_saved(
source.pool_index,
source.set_index,
&checkpoint.info.name,
saved,
);
}
ScannerCheckpointPersistResult::Saved
}
Err(error) => ScannerCheckpointPersistResult::Failed(error),
Err(error) => {
// A successful PUT consumed the old CAS revision. Reload the bucket
// before another write instead of rebasing this scan over a newer writer.
record_checkpoint_save_failure(cache_name, checkpoint, &error);
ScannerCheckpointPersistResult::RetryBucket(error)
}
}
}
fn record_checkpoint_save_failure(cache_name: &str, checkpoint: &DataUsageCache, error: &StorageError) {
global_metrics().record_scanner_checkpoint_save_failed();
error!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "periodic_checkpoint_save_failed",
bucket = %checkpoint.info.name,
cache_name = %cache_name,
error = %error,
"Scanner periodic checkpoint save failed"
);
}
#[derive(Debug)]
pub(crate) struct ScannerCacheLockGuards {
scoped: NamespaceLockGuard,
crates/scanner/src/scanner_io/io_cache.rs
+5 -14
View File
@@ -1209,7 +1209,7 @@ impl ScannerIOCache for SetDisks {
)
.await
{
ScannerCheckpointPersistResult::Saved => {
ScannerCheckpointPersistResult::Saved | ScannerCheckpointPersistResult::Skipped => {
if cache_guard.is_lock_lost() || ctx_clone.is_cancelled() {
scan_ctx.cancel();
await_scanner_disk_shutdown(scan.as_mut()).await;
@@ -1221,19 +1221,10 @@ impl ScannerIOCache for SetDisks {
await_scanner_disk_shutdown(scan.as_mut()).await;
break Err(Error::other("scanner bucket cache fence changed during checkpoint save"));
}
ScannerCheckpointPersistResult::Failed(error) => {
error!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
bucket = %bucket.name,
cache_name = %cache_name,
state = "periodic_checkpoint_save_failed",
error = %error,
"Scanner periodic checkpoint save failed"
);
checkpoint_channel_closed = true;
ScannerCheckpointPersistResult::RetryBucket(error) => {
scan_ctx.cancel();
await_scanner_disk_shutdown(scan.as_mut()).await;
break Err(error);
}
}
}
crates/scanner/src/scanner_io/tests.rs
+170 -1
View File
@@ -717,7 +717,7 @@ async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() {
&mut revisions,
)
.await,
ScannerCheckpointPersistResult::Failed(StorageError::PreconditionFailed)
ScannerCheckpointPersistResult::RetryBucket(StorageError::PreconditionFailed)
),
"matching global fence must not authorize stale cache CAS revisions"
);
@@ -734,6 +734,175 @@ async fn checkpoint_uses_global_cycle_fence_with_set_scoped_cache_revisions() {
assert!(mismatched_sets > 0, "multi-pool fixture must expose a set-scoped cycle fence mismatch");
}
#[derive(Debug)]
struct CheckpointFaultStore {
set: Arc<SetDisks>,
fail_put: AtomicBool,
fail_revision_read: AtomicBool,
}
#[async_trait::async_trait]
impl crate::storage_api::owner::ObjectIO for CheckpointFaultStore {
type Error = crate::EcstoreError;
type RangeSpec = crate::storage_api::owner::HTTPRangeSpec;
type HeaderMap = http::HeaderMap;
type ObjectOptions = crate::ScannerObjectOptions;
type ObjectInfo = crate::ScannerObjectInfo;
type GetObjectReader = crate::ScannerGetObjectReader;
type PutObjectReader = crate::ScannerPutObjReader;
async fn get_object_reader(
&self,
bucket: &str,
object: &str,
range: Option<Self::RangeSpec>,
headers: Self::HeaderMap,
opts: &Self::ObjectOptions,
) -> crate::EcstoreResult<Self::GetObjectReader> {
if self.fail_revision_read.swap(false, Ordering::SeqCst) {
return Err(StorageError::Io(std::io::Error::new(
std::io::ErrorKind::Interrupted,
"injected revision read failure",
)));
}
self.set.get_object_reader(bucket, object, range, headers, opts).await
}
async fn put_object(
&self,
bucket: &str,
object: &str,
data: &mut Self::PutObjectReader,
opts: &Self::ObjectOptions,
) -> crate::EcstoreResult<Self::ObjectInfo> {
if self.fail_put.swap(false, Ordering::SeqCst) {
return Err(StorageError::Lock(LockError::Timeout {
resource: object.to_string(),
timeout: Duration::from_secs(5),
}));
}
self.set.put_object(bucket, object, data, opts).await
}
}
#[async_trait::async_trait]
impl crate::ScannerConfigObjectDelete for CheckpointFaultStore {
async fn delete_config_object(
&self,
bucket: &str,
object: &str,
opts: ScannerObjectOptions,
) -> crate::EcstoreResult<crate::ScannerObjectInfo> {
crate::ScannerConfigObjectDelete::delete_config_object(self.set.as_ref(), bucket, object, opts).await
}
async fn scanner_data_usage_publication_admission(&self) -> Option<crate::ScannerDataUsagePublicationAdmission> {
crate::ScannerConfigObjectDelete::scanner_data_usage_publication_admission(self.set.as_ref()).await
}
}
#[tokio::test]
#[serial]
async fn checkpoint_retries_next_snapshot_after_lock_failure_and_reloads_after_revision_failure() {
let (_temp_dir, fence_store) = setup_local_scanner_store(1, 4).await;
let cycle = 8457;
let leader_epoch = 19;
save_config(
fence_store.clone(),
DATA_USAGE_BLOOM_NAME_PATH.as_str(),
crate::scanner::encode_scanner_cycle_fence_for_test(cycle, leader_epoch),
)
.await
.expect("seed global cycle fence");
let set = fence_store.pools[0].disk_set[0].clone();
let publication_epoch = scanner_publication_epoch(set.clone()).await.expect("read publication epoch");
let cache_name = "checkpoint-recovery/.usage-cache.bin";
DataUsageCache::default()
.save(set.clone(), cache_name)
.await
.expect("seed cache");
let mut revisions = DataUsageCache::read_revisions(set.clone(), cache_name)
.await
.expect("read initial revisions");
let initial_revisions = revisions.clone();
let store = Arc::new(CheckpointFaultStore {
set: set.clone(),
fail_put: AtomicBool::new(true),
fail_revision_read: AtomicBool::new(false),
});
let ctx = CancellationToken::new();
let context = || ScannerCheckpointPersistContext {
ctx: &ctx,
expected_publication_epoch: publication_epoch,
cycle,
leader_epoch,
};
let mut checkpoint = DataUsageCache::default();
checkpoint.info.name = "checkpoint-recovery".to_string();
checkpoint.info.source = Some(DataUsageCacheSource::new(0, 0));
checkpoint.info.scan_checkpoint = Some(crate::DataUsageScanCheckpoint::new(
"checkpoint-recovery/first".to_string(),
crate::DataUsageScanCheckpointReason::Objects,
));
assert!(matches!(
persist_scanner_checkpoint(store.clone(), fence_store.clone(), context(), cache_name, &checkpoint, &mut revisions).await,
ScannerCheckpointPersistResult::Skipped
));
assert_eq!(revisions, initial_revisions);
assert_eq!(
DataUsageCache::read_revisions(set.clone(), cache_name)
.await
.expect("read after failed save"),
initial_revisions
);
checkpoint
.info
.scan_checkpoint
.as_mut()
.expect("checkpoint cursor")
.resume_after = "checkpoint-recovery/second".to_string();
assert!(matches!(
persist_scanner_checkpoint(store.clone(), fence_store.clone(), context(), cache_name, &checkpoint, &mut revisions).await,
ScannerCheckpointPersistResult::Saved
));
assert_ne!(revisions, initial_revisions);
let mut loaded = DataUsageCache::default();
loaded.load(set.clone(), cache_name).await.expect("load recovered checkpoint");
assert_eq!(
loaded.info.scan_checkpoint, checkpoint.info.scan_checkpoint,
"next snapshot must become durable"
);
let saved_revisions = revisions.clone();
checkpoint
.info
.scan_checkpoint
.as_mut()
.expect("checkpoint cursor")
.resume_after = "checkpoint-recovery/third".to_string();
store.fail_revision_read.store(true, Ordering::SeqCst);
assert!(matches!(
persist_scanner_checkpoint(store, fence_store, context(), cache_name, &checkpoint, &mut revisions).await,
ScannerCheckpointPersistResult::RetryBucket(StorageError::Io(_))
));
assert_eq!(revisions, saved_revisions, "indeterminate revision refresh must not rebase the writer");
loaded
.load(set.clone(), cache_name)
.await
.expect("load committed checkpoint after refresh failure");
assert_eq!(
loaded.info.scan_checkpoint, checkpoint.info.scan_checkpoint,
"PUT completed before revision refresh failed"
);
assert_ne!(
DataUsageCache::read_revisions(set, cache_name)
.await
.expect("read committed revisions"),
saved_revisions
);
}
#[derive(Debug)]
struct CheckpointFenceStore {
before: Vec<u8>,
crates/utils/src/http/object_encryption_keys.rs
+34 -5
View File
@@ -55,11 +55,9 @@ pub const MINIO_INTERNAL_ENCRYPTION_KMS_CONTEXT_HEADER: &str = "X-Minio-Internal
/// Reserved RustFS-branded twin of the MinIO-internal SSE key family.
///
/// No RustFS writer emits these keys today — the SSE writer persists the
/// MinIO-branded `X-Minio-Internal-Server-Side-Encryption-*` keys verbatim for
/// interoperability — but redaction (`rustfs_filemeta`) and replication
/// stripping treat the family as sensitive so that a future or third-party
/// writer cannot leak sealed material through the reserved names.
/// The SSE writer persists sealed material under the MinIO-branded names for
/// interoperability. Redaction and replication stripping also cover this twin
/// family, including the encrypted ETag format marker.
pub const RUSTFS_INTERNAL_ENCRYPTION_PREFIX: &str = "x-rustfs-internal-server-side-encryption-";
pub const REPLICATION_SSEC_ALGORITHM_HEADER: &str = "X-Rustfs-Replication-Ssec-Algorithm";
@@ -71,6 +69,15 @@ pub const REPLICATION_SSE_SEAL_ALGORITHM_HEADER: &str = "X-Rustfs-Replication-Se
pub const REPLICATION_SSE_SEALED_KEY_HEADER: &str = "X-Rustfs-Replication-Server-Side-Encryption-Sealed-Key";
pub const REPLICATION_ENCRYPTED_MULTIPART_HEADER: &str = "X-Rustfs-Replication-Encrypted-Multipart";
pub const SUFFIX_OPAQUE_ENCRYPTED_ETAG: &str = "server-side-encryption-opaque-etag";
/// Old encrypted objects can carry a plaintext MD5. Keyless listings expose
/// an encrypted ETag only when its producer recorded the protected format.
pub fn etag_can_be_listed(metadata: &HashMap<String, String>) -> bool {
!metadata.keys().any(|key| is_replication_stripped_encryption_key(key))
|| super::metadata_compat::get_consistent_str(metadata, SUFFIX_OPAQUE_ENCRYPTED_ETAG) == Some("v1")
}
/// Stored SSE-C metadata keys and the wire names they replicate under.
///
/// Source keys must match what `encryption_material_to_metadata` persists; the
@@ -307,6 +314,28 @@ pub fn stored_managed_encryption_key(metadata: &HashMap<String, String>) -> Opti
mod tests {
use super::*;
#[test]
fn keyless_listings_hide_unproven_encrypted_etags() {
assert!(etag_can_be_listed(&HashMap::from([("content-type".to_owned(), "text/plain".to_owned())])));
for key in [
SSEC_ALGORITHM_HEADER,
INTERNAL_ENCRYPTION_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_S3_SEALED_KEY_HEADER,
MINIO_INTERNAL_ENCRYPTION_KMS_DATA_KEY_HEADER,
"X-AMZ-SERVER-SIDE-ENCRYPTION",
"x-rustfs-internal-server-side-encryption-sealed-key",
] {
let mut metadata = HashMap::from([(key.to_owned(), "encryption-metadata".to_owned())]);
assert!(!etag_can_be_listed(&metadata), "legacy encrypted metadata: {key}");
super::super::metadata_compat::insert_str(&mut metadata, SUFFIX_OPAQUE_ENCRYPTED_ETAG, "unknown".to_owned());
assert!(!etag_can_be_listed(&metadata), "unknown producer format: {key}");
super::super::metadata_compat::insert_str(&mut metadata, SUFFIX_OPAQUE_ENCRYPTED_ETAG, "v1".to_owned());
assert!(etag_can_be_listed(&metadata), "protected encrypted metadata: {key}");
metadata.insert("x-minio-internal-server-side-encryption-opaque-etag".to_owned(), "unknown".to_owned());
assert!(!etag_can_be_listed(&metadata), "conflicting format markers: {key}");
}
}
#[test]
fn ssec_transport_projection_retains_only_redacted_reader_headers() {
let mut headers = http::HeaderMap::new();
crates/zip/Cargo.toml
+4
View File
@@ -49,6 +49,10 @@ rustfs-rio.workspace = true
tokio = { workspace = true, features = ["io-util", "macros", "rt"] }
thiserror = { workspace = true }
[target.'cfg(target_os = "macos")'.dependencies]
# Keep XZ decoding self-contained in macOS binaries instead of linking Homebrew liblzma.
liblzma = { version = "0.4.8", features = ["static"] }
[dev-dependencies]
rustfs-tokio-tar = { workspace = true }
futures = { workspace = true }
docs/README.md
+3
View File
@@ -21,6 +21,9 @@ For bounded local read-backend startup and cancellation, see
For legacy protection assessment and bounded protected copies, see [Object integrity inventory, audit, and migration](operations/shard-integrity-audit.md).
For encrypted listing compatibility and older stored fingerprints, see
[Encrypted object ETags](operations/encrypted-etags.md).
Operational runbooks live under [`operations/`](operations/). Replication
operators should start with:
docs/operations/encrypted-etags.md
+44
View File
@@ -0,0 +1,44 @@
# Encrypted object ETags
**Use this when:** upgrading encrypted storage or investigating missing ETags
in object listings after an upgrade.
New encrypted writes use a keyed fingerprint for the ETag rather than the
plaintext MD5. Single-part and part ETags are 32-character hexadecimal values;
completed multipart ETags retain their digest-and-part-count form. They stay
stable for the stored object. Request MD5 and checksum validation still apply
to the original plaintext. Encryption and decryption formats do not change.
## Listings and older objects
ListObjects, ListObjectsV2, and ListObjectVersions omit the optional ETag field
for encrypted objects whose metadata does not attest to the protected format.
This includes encrypted objects written before the upgrade, preserved source
ETags without that proof, and completed encrypted multipart uploads. New
single-part encrypted writes carry the internal format marker and expose their
protected ETag in listings. Unencrypted listings retain their existing ETags.
Existing persistent listing snapshots are invalidated; the provider falls
back to verified storage until a new snapshot is built.
Clients that require an ETag for these older or multipart objects must issue an
authorized HeadObject request, including the SSE-C key when required. GET,
HEAD, and conditional requests continue to use the object's stored ETag.
## Existing data and migration
The upgrade does not rewrite existing object metadata or erase fingerprints
already copied to backups, logs, caches, or replicas. Restrict access to those
copies separately. Rolling back to an older server also restores its listing
behavior, so keep the upgraded listing protection on every serving node.
To obtain a protected ETag, read the object with its encryption credentials and
write it through the upgraded encryption path. Verify the resulting plaintext
and application references before retiring an old copy. A server-side copy
that preserves the source ETag is insufficient.
In a versioned bucket, a new write creates a new version; historical versions
keep their original metadata. Object Lock retention and legal holds still
apply. Do not bypass them or delete retained versions to migrate an ETag.
There is no automatic in-place migration of retained metadata in this upgrade.
Unfinished uploads also keep their original part metadata. Restart an upload
through the upgraded path when its legacy part fingerprints need to be replaced.
docs/operations/oidc-console-integration.md
+2 -2
View File
@@ -1,7 +1,7 @@
# OIDC Console integration
**Use this when:** connecting RustFS Console login to an OpenID Connect provider (Keycloak, Authing, or any standards-compliant IdP), or debugging an OIDC redirect, token, or policy-mapping failure.
**Source of truth:** `crates/config/src/constants/oidc.rs` (provider keys and `RUSTFS_IDENTITY_OPENID_*`), `crates/iam/src/oidc.rs` (discovery, PKCE, token validation, per-provider env suffixes), `rustfs/src/admin/handlers/oidc.rs` (authorize/callback handlers), `crates/config/src/constants/app.rs` (`ENV_RUSTFS_BROWSER_REDIRECT_URL`), `crates/utils/src/egress.rs` (`ENV_OUTBOUND_ALLOW_ORIGINS`), `crates/policy/src/policy/policy.rs` (built-in policies).
**Source of truth:** `crates/config/src/constants/oidc.rs` (provider keys and `RUSTFS_IDENTITY_OPENID_*`), `crates/iam/src/oidc/config.rs` (per-provider env suffixes), `crates/iam/src/oidc/provider.rs` (discovery and JWKS), `crates/iam/src/oidc/runtime.rs` (PKCE and token validation), `rustfs/src/admin/handlers/oidc.rs` (authorize/callback handlers), `crates/config/src/constants/app.rs` (`ENV_RUSTFS_BROWSER_REDIRECT_URL`), `crates/utils/src/egress.rs` (`ENV_OUTBOUND_ALLOW_ORIGINS`), `crates/policy/src/policy/policy.rs` (built-in policies).
The RustFS side is vendor-neutral and is described once; what RustFS requires from any provider is tabulated in [oidc-provider-requirements.md](oidc-provider-requirements.md). The [Keycloak](#keycloak) and [Authing](#authing) sections contain only IdP-side steps and vendor caveats. Examples use provider id `default` and public origin `https://rustfs.example.com`.
@@ -48,7 +48,7 @@ Process-level settings (environment only, never suffixed per provider):
| `RUSTFS_BROWSER_REDIRECT_URL` | Public browser origin used for callback generation, Console success redirects, and logout fallback. |
| `RUSTFS_OUTBOUND_ALLOW_ORIGINS` | Exact `scheme://host[:port]` origins RustFS may contact for discovery, JWKS, and token requests when the IdP resolves to a private, loopback, or container-network address. See [outbound-connection-policy.md](outbound-connection-policy.md). |
Named providers: to use provider id `<id>`, suffix every provider env var with `_<id>` (for example `RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak`) and register the callback `/rustfs/admin/v3/oidc/callback/<id>`. Suffix scanning is `parse_single_provider` in `crates/iam/src/oidc.rs`.
Named providers: to use provider id `<id>`, suffix every provider env var with `_<id>` (for example `RUSTFS_IDENTITY_OPENID_CLIENT_ID_keycloak`) and register the callback `/rustfs/admin/v3/oidc/callback/<id>`. Suffix scanning is `parse_single_provider` in `crates/iam/src/oidc/config.rs`.
Restart RustFS after changing any of these settings.
docs/operations/oidc-provider-requirements.md
+10 -10
View File
@@ -1,7 +1,7 @@
# OIDC provider requirements
**Use this when:** evaluating whether an identity provider (or a vendor's "OIDC-like" OAuth product) can back RustFS console SSO, or debugging why a provider fails discovery, login, or policy mapping.
**Source of truth:** `crates/iam/src/oidc.rs` (`OidcProviderConfig`, `discover_provider`, `trusted_aud`, `map_claims_to_policies`); `crates/config/src/constants/oidc.rs` (`OIDC_DEFAULT_*`, `ENV_IDENTITY_OPENID_*`); `rustfs/src/admin/handlers/oidc.rs` (`derive_callback_uri_with_provider_config`, `browser_redirect_url`).
**Source of truth:** `crates/iam/src/oidc/config.rs` (`OidcProviderConfig`), `crates/iam/src/oidc/provider.rs` (`discover_provider`), `crates/iam/src/oidc/runtime.rs` (`trusted_aud`, `map_claims_to_policies`); `crates/config/src/constants/oidc.rs` (`OIDC_DEFAULT_*`, `ENV_IDENTITY_OPENID_*`); `rustfs/src/admin/handlers/oidc.rs` (`derive_callback_uri_with_provider_config`, `browser_redirect_url`).
RustFS is a standard OpenID Connect relying party using the authorization-code flow with PKCE. It reads every claim it uses from the ID token; it never calls the UserInfo endpoint. Provider-side setup steps for Keycloak and Authing, and the RustFS-side configuration keys, are in [oidc-console-integration.md](oidc-console-integration.md).
@@ -11,16 +11,16 @@ The table below describes Console login. For endpoint-free STS issuers, use the
| # | Requirement | Details | Code anchor |
| --- | --- | --- | --- |
| 1 | Discovery document | `RUSTFS_IDENTITY_OPENID_CONFIG_URL` names the provider (issuer base or full discovery URL); RustFS fetches `{issuer}/.well-known/openid-configuration` and needs `issuer`, `authorization_endpoint`, `token_endpoint`, `jwks_uri`, and the standard `*_supported` arrays. When `RUSTFS_IDENTITY_OPENID_ISSUER` is set, the document's `issuer` must equal it exactly; otherwise RustFS tries the issuer candidates derived from the config URL. | `crates/iam/src/oidc.rs` `discover_provider`, `discover_provider_from_config_url` |
| 2 | Outbound reachability | Discovery, JWKS, and token requests go through the shared egress policy. A provider on a private or loopback address needs its exact origin in `RUSTFS_OUTBOUND_ALLOW_ORIGINS`; otherwise startup logs `OIDC provider discovery blocked by outbound policy`. | `crates/iam/src/oidc.rs` `OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY`; [Outbound connection policy](outbound-connection-policy.md) |
| 3 | Signed ID token, verifiable via JWKS | The ID token signature is verified against `jwks_uri`; the key set is refreshed after `OIDC_JWKS_REFRESH_INTERVAL` and once more on a verification failure. A token response without `id_token` fails login — an OAuth-only access token is not sufficient. | `crates/iam/src/oidc.rs` `OIDC_JWKS_REFRESH_INTERVAL`, the `no id_token in token response` error |
| 4 | Authorization-code flow with PKCE (S256) | The authorization request carries `response_type=code`, `scope`, `redirect_uri`, `state`, `nonce`, `code_challenge`, `code_challenge_method=S256`; the token request carries the `code_verifier`. Providers that ignore or reject PKCE, `state`, or `nonce` are not supported. | `crates/iam/src/oidc.rs` `PkceCodeChallenge::new_random_sha256`; `crates/iam/src/oidc_state.rs` `pkce_verifier` |
| 5 | Scopes | Default `openid,profile,email` (`OIDC_DEFAULT_SCOPES`); override with `RUSTFS_IDENTITY_OPENID_SCOPES` (comma-separated). The provider must accept every configured scope. | `crates/config/src/constants/oidc.rs` `OIDC_DEFAULT_SCOPES`; `crates/iam/src/oidc.rs` `OidcProviderConfig::scopes` |
| 6 | Callback returns `state` | The callback must carry both `code` and the original `state`; `state` locates the in-flight session that holds the nonce and PKCE verifier. A callback with only `code` fails. | `rustfs/src/admin/handlers/oidc.rs` `OIDC_CALLBACK_SUFFIX`; `crates/iam/src/oidc_state.rs` |
| 7 | ID token claims | `iss`, `aud`, `exp`, and `nonce` are verified; `sub` identifies the user. `aud` must contain the client id or one of `RUSTFS_IDENTITY_OPENID_OTHER_AUDIENCES`. | `crates/iam/src/oidc.rs` `trusted_aud`, `OidcClaims` |
| 8 | Authorization claims in the ID token | Policy mapping reads the claim named by `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` (default `groups`, `OIDC_DEFAULT_CLAIM_NAME`) together with `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` and the optional `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM`, as a string or array of strings, optionally prefixed by `RUSTFS_IDENTITY_OPENID_CLAIM_PREFIX`. Values must equal RustFS policy names (for example `consoleAdmin`, `readwrite`, `readonly`) made only of `[A-Za-z0-9_:.-]`. Groups-claim and roles-claim values with no policy of that name, such as the Active Directory group `DOMAIN\Domain Users` or Keycloak's `offline_access` role, are ignored once storage confirms that no such policy exists. Login is rejected when a policy lookup fails, when no value resolves, when a value names an existing policy whose name is outside that character set (dropping it could remove a Deny), or when a name from `RUSTFS_IDENTITY_OPENID_ROLE_POLICY`, a separate `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` claim, or an explicit IAM policy mapping does not exist. Email and username come from `RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM` (default `email`) and `RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM` (default `preferred_username`). A provider that returns only a user id can authenticate but cannot express RustFS authorization. | `crates/iam/src/oidc.rs` `map_claims_to_policies`, `extract_canonical_group_values`; `OidcSys::group_claim_policy_names`; `rustfs/src/admin/service/federated_identity.rs` `select_oidc_policies` |
| 1 | Discovery document | `RUSTFS_IDENTITY_OPENID_CONFIG_URL` names the provider (issuer base or full discovery URL); RustFS fetches `{issuer}/.well-known/openid-configuration` and needs `issuer`, `authorization_endpoint`, `token_endpoint`, `jwks_uri`, and the standard `*_supported` arrays. When `RUSTFS_IDENTITY_OPENID_ISSUER` is set, the document's `issuer` must equal it exactly; otherwise RustFS tries the issuer candidates derived from the config URL. | `crates/iam/src/oidc/provider.rs` `discover_provider`, `discover_provider_from_config_url` |
| 2 | Outbound reachability | Discovery, JWKS, and token requests go through the shared egress policy. A provider on a private or loopback address needs its exact origin in `RUSTFS_OUTBOUND_ALLOW_ORIGINS`; otherwise startup logs `OIDC provider discovery blocked by outbound policy`. | `crates/iam/src/oidc/provider.rs` `OIDC_DISCOVERY_BLOCKED_BY_OUTBOUND_POLICY`; [Outbound connection policy](outbound-connection-policy.md) |
| 3 | Signed ID token, verifiable via JWKS | The ID token signature is verified against `jwks_uri`; the key set is refreshed after `OIDC_JWKS_REFRESH_INTERVAL` and once more on a verification failure. A token response without `id_token` fails login — an OAuth-only access token is not sufficient. | `crates/iam/src/oidc/provider.rs` `OIDC_JWKS_REFRESH_INTERVAL`; `crates/iam/src/oidc/runtime.rs` `no id_token in token response` |
| 4 | Authorization-code flow with PKCE (S256) | The authorization request carries `response_type=code`, `scope`, `redirect_uri`, `state`, `nonce`, `code_challenge`, `code_challenge_method=S256`; the token request carries the `code_verifier`. Providers that ignore or reject PKCE, `state`, or `nonce` are not supported. | `crates/iam/src/oidc/runtime.rs` `PkceCodeChallenge::new_random_sha256`; `crates/iam/src/oidc/state.rs` `pkce_verifier` |
| 5 | Scopes | Default `openid,profile,email` (`OIDC_DEFAULT_SCOPES`); override with `RUSTFS_IDENTITY_OPENID_SCOPES` (comma-separated). The provider must accept every configured scope. | `crates/config/src/constants/oidc.rs` `OIDC_DEFAULT_SCOPES`; `crates/iam/src/oidc/config.rs` `OidcProviderConfig::scopes` |
| 6 | Callback returns `state` | The callback must carry both `code` and the original `state`; `state` locates the in-flight session that holds the nonce and PKCE verifier. A callback with only `code` fails. | `rustfs/src/admin/handlers/oidc.rs` `OIDC_CALLBACK_SUFFIX`; `crates/iam/src/oidc/state.rs` |
| 7 | ID token claims | `iss`, `aud`, `exp`, and `nonce` are verified; `sub` identifies the user. `aud` must contain the client id or one of `RUSTFS_IDENTITY_OPENID_OTHER_AUDIENCES`. | `crates/iam/src/oidc/runtime.rs` `trusted_aud`, `OidcClaims` |
| 8 | Authorization claims in the ID token | Policy mapping reads the claim named by `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` (default `groups`, `OIDC_DEFAULT_CLAIM_NAME`) together with `RUSTFS_IDENTITY_OPENID_GROUPS_CLAIM` and the optional `RUSTFS_IDENTITY_OPENID_ROLES_CLAIM`, as a string or array of strings, optionally prefixed by `RUSTFS_IDENTITY_OPENID_CLAIM_PREFIX`. Values must equal RustFS policy names (for example `consoleAdmin`, `readwrite`, `readonly`) made only of `[A-Za-z0-9_:.-]`. Groups-claim and roles-claim values with no policy of that name, such as the Active Directory group `DOMAIN\Domain Users` or Keycloak's `offline_access` role, are ignored once storage confirms that no such policy exists. Login is rejected when a policy lookup fails, when no value resolves, when a value names an existing policy whose name is outside that character set (dropping it could remove a Deny), or when a name from `RUSTFS_IDENTITY_OPENID_ROLE_POLICY`, a separate `RUSTFS_IDENTITY_OPENID_CLAIM_NAME` claim, or an explicit IAM policy mapping does not exist. Email and username come from `RUSTFS_IDENTITY_OPENID_EMAIL_CLAIM` (default `email`) and `RUSTFS_IDENTITY_OPENID_USERNAME_CLAIM` (default `preferred_username`). A provider that returns only a user id can authenticate but cannot express RustFS authorization. | `crates/iam/src/oidc/runtime.rs` `map_claims_to_policies`, `extract_canonical_group_values`, `OidcSys::group_claim_policy_names`; `rustfs/src/admin/service/federated_identity.rs` `select_oidc_policies` |
| 9 | Registered redirect URI | The provider must accept the callback `{public-origin}/rustfs/admin/v3/oidc/callback/{provider_id}`. RustFS picks the origin in this order: the provider's `redirect_uri` (`RUSTFS_IDENTITY_OPENID_REDIRECT_URI`), then `RUSTFS_BROWSER_REDIRECT_URL`, then the request's own scheme and host — the last only when `RUSTFS_IDENTITY_OPENID_REDIRECT_URI_DYNAMIC` is enabled. | `rustfs/src/admin/handlers/oidc.rs` `derive_callback_uri_with_provider_config`, `browser_redirect_url` |
| 10 | Logout endpoint (optional) | When discovery advertises `end_session_endpoint`, RustFS builds an RP-initiated logout URL with `id_token_hint`, `client_id`, and `post_logout_redirect_uri`. Without it, logout falls back to the console login page. | `crates/iam/src/oidc.rs` `build_logout_url` (reads `end_session_endpoint` from `ProviderMetadataWithLogout`) |
| 10 | Logout endpoint (optional) | When discovery advertises `end_session_endpoint`, RustFS builds an RP-initiated logout URL with `id_token_hint`, `client_id`, and `post_logout_redirect_uri`. Without it, logout falls back to the console login page. | `crates/iam/src/oidc/runtime.rs` `build_logout_url` (reads `end_session_endpoint` from `ProviderMetadataWithLogout`) |
## STS workload providers
docs/operations/outbound-connection-policy.md
+1 -1
View File
@@ -16,7 +16,7 @@ RustFS validates every operator-configured outbound destination to close a serve
| --- | --- | --- |
| Event-notification webhooks (`RUSTFS_NOTIFY_WEBHOOK_*`) and audit webhooks (`RUSTFS_AUDIT_WEBHOOK_*`) | Full policy | Proxies disabled and redirects not followed, so the endpoint must be reachable directly (`crates/targets/src/target/webhook.rs`) |
| Target configuration validation (startup and admin API) | Full policy | `crates/targets/src/config/common.rs` `validate_outbound_http_url`; `rustfs/src/admin/handlers/target_descriptor.rs` |
| OIDC discovery, JWKS, and token requests | Full policy | A blocked provider logs `OIDC provider discovery blocked by outbound policy` naming the origin to allowlist (`crates/iam/src/oidc.rs`) |
| OIDC discovery, JWKS, and token requests | Full policy | A blocked provider logs `OIDC provider discovery blocked by outbound policy` naming the origin to allowlist (`crates/iam/src/oidc/provider.rs`, `crates/iam/src/oidc/transport.rs`) |
| Object Lambda targets | Full policy | `rustfs/src/admin/router.rs` `outbound_policy` |
| Bucket replication targets | Literal check, relaxed | Private addresses are always allowed; loopback only with `RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET=true` (`crates/ecstore/src/bucket/remote_s3_client.rs` `validate_remote_endpoint`, shared with on-demand migration sources) |
| On-demand migration sources | Literal check, relaxed | Same guard and same escape hatch as replication targets (see below) |
docs/operations/scanner-runtime-controls.md
+14
View File
@@ -172,6 +172,20 @@ catch_up_estimate.discovered_expiry_items
catch_up_estimate.discovered_transition_items
```
### Periodic checkpoint persistence
`metrics.scan_checkpoint` is the in-memory resume hint; it does not confirm a durable save. `metrics.scan_checkpoints_persisted` lists the latest confirmed periodic save for each `(pool_index, set_index)` handled by this process, with its bucket, `saved_unix_secs`, and optional checkpoint cursor. It retains one observation per set, is reset on process restart, and does not include the final bucket save. Query the peer executing the walk for remote-worker observations. `metrics.scan_checkpoint_save_failures` counts failed periodic saves, including failures to refresh CAS revisions after a write; failed attempts do not advance the confirmed save observation.
An ordinary failed save drops that snapshot and leaves the bounded checkpoint receiver active, so the next producer interval tries again. A CAS conflict or an unreadable revision after a successful write cancels the bucket attempt and reloads persisted state through the existing retry path. Leader, cycle, and publication fencing remain mandatory. This preserves future periodic save attempts for a continuing bucket walk.
Periodic snapshots include the suspended ancestor accounting needed to connect a nested active branch to the bucket root. Completed coverage is sealed independently of raw enumeration bookkeeping. A valid coverage frontier can therefore skip completed subtrees after a fenced leader handoff without counting unfinished work as completed. Emission remains bounded by the existing interval/object thresholds and capacity-one channel; invalid snapshot attempts also consume the producer interval so they cannot repeatedly clone the cache on every object.
A resumed walk retains its previous durable coverage while initial enumeration has not recopied the covered subtrees. Unfinished out-of-order heal revisits cannot be certified as completed prefix work, including in cancellation snapshots.
Shared compacted accumulators defer periodic emission until the owning subtree completes; replay inside that compacted subtree remains possible after an interruption. When bucket checkpoint preparation returns `Reset` for existing resume state, it logs `checkpoint_reset` with the validation reason. The `scan_checkpoint_used` counter includes validated coverage adoption by the forward sweep.
A confirmed periodic save does not prove restart resumability. Adoption still validates bucket/set identity and, for non-empty caches, connected cache scope. Skipping completed subtrees additionally requires a validated coverage frontier; raw enumeration state alone does not prove completed subtree coverage.
## Usage State Reset
The supported break-glass route for rebuilding scanner usage state is `POST /v3/scanner/usage-state/reset` with body `{"mode":"full-rebuild"}`, authenticated as an admin identity holding `ConfigUpdateAdminAction` (route registered in `rustfs/src/admin/route_registration_test.rs`). Use it only after the scanner status shows a usage-floor load failure, a conflicting persisted usage floor, or an operator decision to discard the durable usage baseline and rebuild it from a full scanner pass.
docs/testing/ci-gates.md
+14 -7
View File
@@ -15,10 +15,16 @@ The `main` ruleset (`6436880`) requires exactly these contexts, with `strict_req
| `Quick Checks` | `ci.yml` job `quick-checks` | Formatting and repository guard scripts |
| `Test and Lint` | `ci.yml` job `required-checks` | Exact expected results for every CI validation job, including workspace checks, critical E2E, feature lanes, and event-specific full suites |
Every PR enters `ci.yml`. The `classify-changes` job uses the base revision of `scripts/ci_gate.py` to select a conservative documentation-only path: root Markdown/licenses, `AGENTS.md`, Markdown under `docs/` or `.agents/skills/`, and documentation images. Unknown paths, unavailable Git history, an empty diff, or a missing base policy select the full matrix. Renames include their deleted source path. Documentation-only PRs still run Quick Checks and Typos; the aggregate requires the expensive jobs to be skipped exactly as selected.
Every PR enters `ci.yml`. The `classify-changes` job uses the base revision of `scripts/ci_gate.py` to select a conservative documentation-only path: root Markdown/licenses, `AGENTS.md`, Markdown under `docs/` or `.agents/skills/`, and documentation images. Unknown paths, unavailable Git history, an empty diff, or a missing base policy select the full scope for that event. Renames include their deleted source path. Documentation-only PRs still run Quick Checks and Typos; the aggregate requires the expensive jobs to be skipped exactly as selected.
PRs changing only Rust sources under `crates/e2e_test/src/` and known E2E selection digests (optionally with those documentation paths) use the `e2e` scope. They keep workspace validation, the server build, E2E smoke, and both S3 lanes; independent workflows retain their path selection. Connect boundary, serial ILM, optional protocol/rio-v2, and real io_uring jobs are skipped because their sources and test inputs are unchanged. Manifests, fixtures, shared nextest configuration, scripts, production sources, and unknown paths still select `full`. A dependency on `e2e_test` from another workspace member or reachable local dependency disables this shortcut. All non-PR events retain their full selection.
Alternate-feature unit tests and Clippy for rio-v2, Swift, SFTP, and HTTP/3 run on main and `gateway/integration` pushes, merge groups, and scheduled/manual validation. PRs run the default workspace and retain ILM, S3, real io_uring, and Connect security boundaries in their applicable scopes. The aggregate requires the alternate-feature jobs to be skipped on PRs and successful on non-PR events. This reduces repeated builds across PR revisions; feature-specific regressions can be detected after merging when the merge queue is not used. For changes to these features, dispatch `ci.yml` against the candidate branch before merging when that signal is needed. Scheduled results supplement the main baseline and do not replace it.
Workspace Clippy uses the same non-PR cadence. Every code PR still builds and runs the complete default workspace test suite, documentation tests, and migration proofs; Quick Checks retain formatting and script/workflow linting. Without a merge queue, Clippy diagnostics may first fail main after merge. A manual `ci.yml` dispatch against the candidate branch runs Clippy before merging when needed.
Serial ILM tests run at the end of `test-and-lint` for the `full` scope, reusing the workspace's Cargo package and feature selection. The existing ignored-test filter and `-j1` remain; `--profile default` keeps the no-retry execution and preserves the separate core CI JUnit report. Any ILM failure fails the workspace job and required aggregate. Logs remain in the `ilm-integration` artifact. There is no separate ILM runner or cache restore.
`required-checks` runs even after failed or skipped dependencies. `scripts/ci_gate.py verify` rejects missing jobs, unexpected jobs, failure, cancellation, and unexpected skips; optional lanes are required only on their declared events. `Workspace Test and Lint` is the ordinary Rust job, while `Test and Lint` uniquely names the aggregate. New validation jobs must update both its direct dependencies and the script contract. A report-only lane is registered in `REPORT_ONLY_JOBS` instead: it stays out of the aggregate, must keep its distinguishing environment, and must keep the steps it reruns identical to the required job it mirrors. Test this wiring and its failure cases with `python3 scripts/ci_gate.py --self-test`.
The `gateway/integration` ruleset (`24663286`) requires the same three contexts with the same review rules for the s3s to gateway integration branch (rustfs/backlog#2734). It omits `required_linear_history` because `gateway-integration-sync.yml` lands `--no-ff` merge commits from `main` on that branch every weekday, and it names the `rustfs-cla-assistant` app as the bypass actor that pushes them. `ci.yml` runs on pull requests targeting either branch and on pushes to either branch; a push to `gateway/integration` also runs `e2e-full`, because the daily merge commit lands without a pull request and must be proven like a `main` push.
@@ -45,10 +51,11 @@ The four site-replication state-writer concurrency proofs reserve the available
| Event | Check name | Workflow / job | Merge status | Reproduce |
|---|---|---|---|---|
| PR, non-doc change | `Quick Checks` | `ci.yml` `quick-checks` | Required | `make pre-commit` |
| PR, non-doc change | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo clippy --all-targets -- -D warnings`; `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --doc`; migration evidence check described below |
| PR, non-doc change | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate | `cargo nextest run --profile ci --all --exclude e2e_test`; `cargo test --all --exclude e2e_test --doc`; migration evidence check described below |
| Main push, merge group, schedule, manual dispatch | `Workspace Test and Lint` | `ci.yml` `test-and-lint` | Via aggregate on these events | workspace tests plus `cargo clippy --all-targets -- -D warnings` |
| PR, non-doc change | `Typos` | `ci.yml` `typos` | Via aggregate | `typos` |
| PR, full selection | `ILM Integration (serial)` | `ci.yml` `test-ilm-integration-serial` | Via aggregate | exact command in the job |
| PR, full selection | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate | `cargo nextest run` with the job's `--features` |
| Full selection | `Workspace Test and Lint` (serial ILM step) | `ci.yml` `test-and-lint` | Via aggregate | exact `Run ignored ILM integration tests serially` command in the job |
| Main push, merge group, schedule, manual dispatch | `Test and Lint (rio-v2)`, `Test and Lint (swift)`, `Test and Lint (sftp)`, `Test and Lint (http3)` | `ci.yml` `test-and-lint-rio-v2`, `test-and-lint-protocols` | Via aggregate on these events; skipped on PRs | `cargo nextest run` with the job's `--features` |
| PR, full selection | `Connect Short Credential Boundary` | `ci.yml` `connect-short-credential-boundary` | Via aggregate | `cargo test -p rustfs --test connect_registration --features connect-e2e-short-credentials`; `cargo check -p rustfs --release --features connect-e2e-short-credentials` must fail |
| PR, full selection | `Offline Enrollment E2E Root Boundary` | `ci.yml` `offline-enrollment-e2e` | Via aggregate | `scripts/check_offline_enrollment_e2e.sh` |
| PR, non-doc change | `Build RustFS Debug Binary` | `ci.yml` `build-rustfs-debug-binary` | Via aggregate; prerequisite for black-box jobs | `python3 scripts/e2e_binary.py build --bins --features e2e-test-hooks` (binary plus its `rustfs.e2e.json` sidecar) |
@@ -62,7 +69,7 @@ The four site-replication state-writer concurrency proofs reserve the available
| PR touching `paths` in `fuzz.yml` | `Build Fuzz Harness`, `Smoke / <target>` | `fuzz.yml` `fuzz-build`, `pr-fuzz-smoke` | Report-only | `MAX_TOTAL_TIME=60 ./scripts/fuzz/run.sh` |
| PR touching `paths` in `windows-filesystem.yml` | `Rename Safety` | `windows-filesystem.yml` `rename-safety` | Report-only | the `cargo test -p rustfs-ecstore --lib <filter>` commands in the job, on Windows |
| PR touching `paths` in `coverage.yml` | `Workspace line coverage` | `coverage.yml` `coverage` | Report-only | `make coverage`; `python3 scripts/check_security_coverage.py target/llvm-cov/coverage.json` |
| PR touching `paths` in `e2e-upgrade.yml` | `Direct upgrade from the previous release`, `Mixed-version rolling upgrade from the previous release`, `Bucket configuration survives the upgrade`, `Rollback reads current bucket metadata` | `e2e-upgrade.yml` `upgrade` matrix | Report-only | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` command with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) |
| PR touching `paths` in `e2e-upgrade.yml` | `Upgrade compatibility` | `e2e-upgrade.yml` `upgrade` | Report-only | `python3 scripts/e2e_binary.py build`, then the job's `python3 scripts/e2e_binary.py run -- cargo test --locked -p e2e_test` commands with `RUSTFS_UPGRADE_SOURCE_BINARY` pointing at the pinned previous release (`UPGRADE_SOURCE_VERSION`) |
| PR touching `paths` in `oidc-keycloak.yml` | `OIDC Keycloak live gate` | `oidc-keycloak.yml` `oidc-keycloak-live` | Report-only | `cargo build --locked -p rustfs --bin rustfs`, then `bash scripts/test/oidc_keycloak_live.sh ./target/debug/rustfs` |
| PR touching `paths` in `targets-integration.yml` | `PostgreSQL, MySQL, AMQP, and NATS` | `targets-integration.yml` `targets-live` | Report-only | start the containers as in the job, export the `RUSTFS_TEST_*` DSNs, then the job's `cargo test --locked -p rustfs-targets --test <name> -- --ignored --test-threads=1` commands |
| PR, documentation-only selection | `Quick Checks`, `Typos`, `Test and Lint` | `ci.yml` `quick-checks`, `typos`, `required-checks` | Required directly or via aggregate | Quick Checks commands; `python3 scripts/ci_gate.py --self-test` |
@@ -70,7 +77,7 @@ The four site-replication state-writer concurrency proofs reserve the available
e2e filters live in `.config/nextest.toml`; extend a profile instead of adding a second selector. Before a profile runs, `scripts/check_test_wiring.py` compares its listing to the committed digest in `.config/e2e-<profile>-selection.txt`, so a silent test drop fails closed.
Architecture migration rules run once in the required Quick Checks job. Migration proofs reuse the successful workspace run: `scripts/check_migration_gate_count.sh evidence <core-listing.json> <junit.xml>` checks the unchanged name selection and committed floor, and requires exactly one successful execution without retries for each proof. Missing, filtered, skipped, failed, or duplicate evidence fails the gate. Local `check` and `run` modes remain available. Upgrade compatibility builds one current server and shares its binary and provenance sidecar across the seven cases; each case independently verifies the pinned previous release checksum and runs its existing test.
Architecture migration rules run once in the required Quick Checks job. Migration proofs reuse the successful workspace run: `scripts/check_migration_gate_count.sh evidence <core-listing.json> <junit.xml>` checks the unchanged name selection and committed floor, and requires exactly one successful execution without retries for each proof. Missing, filtered, skipped, failed, or duplicate evidence fails the gate. Local `check` and `run` modes remain available. Upgrade compatibility builds one current server and shares its binary and provenance sidecar with one test runner. That runner verifies the pinned previous release checksum once, reuses the compiled harness, and executes every existing upgrade case in a separate process. Failed cases do not stop the remaining cases; any failure fails the job. Logs and G09 evidence retain per-case subdirectories in the aggregate artifacts. `scripts/test_upgrade_workflow.py` checks case membership and failure propagation.
Scanner usage and heal rebuild coverage are intentionally split by risk and
cost. `data_usage_test` runs in the PR `e2e-smoke` lane so changes that affect
@@ -100,7 +107,7 @@ Scheduled lanes never block a PR. Their workflow-local gate fails the run, sched
| `mint.yml` (weekly) | `mint` | report-only by design; per-suite PASS/FAIL/NA and raw `log.json` | yes | pinned Docker sequence in the workflow |
| `coverage.yml` (weekly) | `coverage` | report-only trend; lcov and JSON artifact | yes | `make coverage` |
| `runner-hygiene.yml` (monthly) | `check-ephemerality` | runner ephemerality | yes | dispatch |
| `e2e-upgrade.yml` (weekly) | `upgrade` (4-case matrix) | upgrade and rollback gate; server logs | no | see the PR row |
| `e2e-upgrade.yml` (weekly) | `upgrade` (serial cases sharing one harness build) | upgrade and rollback gate; server logs | no | see the PR row |
| `oidc-keycloak.yml` (weekly) | `oidc-keycloak-live` | live OIDC gate | no | see the PR row |
| `targets-integration.yml` (nightly) | `targets-live` | live target gate; container logs | no | see the PR row |
| `gateway-integration-sync.yml` (weekdays) | `sync` | merges `main` into `gateway/integration` as a `--no-ff` merge commit; a conflicting merge is aborted and opens a `sync/<source>-<date>` pull request listing the conflicting files, which keeps the run green; a rejected push or missing pull request fails it | no | dispatch, optionally with `source_ref` |
rustfs/Cargo.toml
+1
View File
@@ -389,6 +389,7 @@ symbolic-demangle = { version = "=13.9.0", default-features = false, features =
[dev-dependencies]
uuid = { workspace = true, features = ["v4", "v5", "fast-rng", "macro-diagnostics"] }
rustfs-iam = { workspace = true, features = ["test-util"] }
serial_test = { workspace = true }
tempfile = { workspace = true }
aws-config = { workspace = true }
rustfs/src/admin/handlers/bucket_recovery.rs
+11 -13
View File
@@ -17,13 +17,12 @@
use crate::admin::auth::authorize_admin_request;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::runtime_sources::current_object_store_handle;
use crate::admin::storage_api::s3::{Body, S3Error, S3Request, S3Response, S3Result};
use crate::admin::storage_api::s3::{self, Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result};
use crate::server::ADMIN_PREFIX;
use crate::site_replication::{site_replication_enabled, with_site_replication_bucket_mutation_lock};
use hyper::{Method, StatusCode};
use matchit::Params;
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_s3_types::s3_error;
use serde::{Deserialize, Serialize};
use tracing::info;
use uuid::Uuid;
@@ -64,23 +63,23 @@ impl Operation for RecoverOrphanedBucketHandler {
.credentials
.as_ref()
.map(|credentials| credentials.secret_key.clone())
.ok_or_else(|| s3_error!(InvalidRequest, "authentication required"))?;
.ok_or_else(|| s3::error(S3ErrorCode::InvalidRequest, "authentication required"))?;
let bucket = params
.get("bucket")
.filter(|bucket| !bucket.is_empty())
.ok_or_else(|| s3_error!(InvalidRequest, "bucket name is required"))?;
.ok_or_else(|| s3::error(S3ErrorCode::InvalidRequest, "bucket name is required"))?;
let body = req
.input
.store_all_limited(rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE)
.await
.map_err(|_| s3_error!(InvalidRequest, "failed to read recovery request body"))?;
.map_err(|_| s3::error(S3ErrorCode::InvalidRequest, "failed to read recovery request body"))?;
let request: RecoverOrphanedBucketRequest = serde_json::from_slice(&body)
.map_err(|_| s3_error!(InvalidRequest, "expected JSON with a non-nil expectedIncarnationId"))?;
.map_err(|_| s3::error(S3ErrorCode::InvalidRequest, "expected JSON with a non-nil expectedIncarnationId"))?;
if request.expected_incarnation_id.is_nil() {
return Err(s3_error!(InvalidRequest, "expectedIncarnationId must be non-nil").into());
return Err(s3::error(S3ErrorCode::InvalidRequest, "expectedIncarnationId must be non-nil"));
}
let Some(store) = current_object_store_handle() else {
return Err(s3_error!(InternalError, "object store is not initialized").into());
return Err(s3::error(S3ErrorCode::InternalError, "object store is not initialized"));
};
let operation_bucket = bucket.to_owned();
@@ -92,11 +91,10 @@ impl Operation for RecoverOrphanedBucketHandler {
let lock_bucket = operation_bucket.clone();
with_site_replication_bucket_mutation_lock(lock_store, &lock_bucket, move || async move {
if site_replication_enabled().await? {
return Err(s3_error!(
OperationAborted,
"orphaned bucket recovery is local; reconcile every site before recreating this bucket"
)
.into());
return Err(s3::error(
S3ErrorCode::OperationAborted,
"orphaned bucket recovery is local; reconcile every site before recreating this bucket",
));
}
operation_store
.recover_orphaned_bucket(&operation_bucket, expected_incarnation)
rustfs/src/admin/handlers/oidc.rs
+57 -654
View File
@@ -13,36 +13,22 @@
// limitations under the License.
use crate::admin::auth::authorize_admin_request;
use crate::admin::handlers::supervise_admin_mutation;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::runtime_sources::{
current_app_context, current_federated_identity_service, current_object_store_handle_for_context,
current_server_config_for_context,
};
use crate::admin::runtime_sources::current_federated_identity_service;
use crate::admin::service::federated_identity::DefaultFederatedSessionBinding;
use crate::admin::storage_api::config::{
read_admin_config_without_migrate, read_admin_server_config_snapshot, save_admin_server_config_snapshot,
};
use crate::admin::service::oidc_config;
use crate::admin::utils::json_response;
use crate::server::{ADMIN_PREFIX, MINIO_ADMIN_PREFIX, console_prefix};
use http::StatusCode;
use hyper::Method;
use matchit::Params;
use rustfs_config::oidc::{
IDENTITY_OPENID_SUB_SYS, OIDC_CLAIM_NAME, OIDC_CLAIM_PREFIX, OIDC_CLIENT_ID, OIDC_CLIENT_SECRET, OIDC_CONFIG_URL,
OIDC_DEFAULT_CLAIM_NAME, OIDC_DEFAULT_EMAIL_CLAIM, OIDC_DEFAULT_GROUPS_CLAIM, OIDC_DEFAULT_ROLES_CLAIM, OIDC_DEFAULT_SCOPES,
OIDC_DEFAULT_USERNAME_CLAIM, OIDC_DISPLAY_NAME, OIDC_EMAIL_CLAIM, OIDC_GROUPS_CLAIM, OIDC_HIDE_FROM_UI, OIDC_ISSUER,
OIDC_OTHER_AUDIENCES, OIDC_REDIRECT_URI, OIDC_REDIRECT_URI_DYNAMIC, OIDC_ROLE_POLICY, OIDC_ROLES_CLAIM, OIDC_SCOPES,
OIDC_USERNAME_CLAIM,
};
use rustfs_config::server_config::Config as ServerConfig;
use rustfs_config::{DEFAULT_DELIMITER, ENABLE_KEY, ENV_RUSTFS_BROWSER_REDIRECT_URL, EnableState, MAX_ADMIN_REQUEST_BODY_SIZE};
use rustfs_config::{ENV_RUSTFS_BROWSER_REDIRECT_URL, MAX_ADMIN_REQUEST_BODY_SIZE};
use rustfs_iam::federation::{FederatedSessionBindingError, FederationError};
use rustfs_iam::oidc::{OidcProviderConfigInput, OidcProviderValidationInput};
use rustfs_policy::policy::action::{Action, AdminAction};
use rustfs_utils::egress::validate_outbound_url;
use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
use serde::Serialize;
use serde::de::DeserializeOwned;
use serde::{Deserialize, Serialize};
use time::OffsetDateTime;
use tracing::{debug, error, warn};
use url::Url;
@@ -78,6 +64,19 @@ fn callback_federation_error(error: FederationError) -> S3Error {
}
}
fn config_service_error(error: oidc_config::OidcAdminConfigError) -> S3Error {
use oidc_config::OidcAdminConfigError;
let code = match &error {
OidcAdminConfigError::Config(rustfs_iam::oidc::OidcConfigError::EnvironmentManaged) => S3ErrorCode::AccessDenied,
OidcAdminConfigError::Config(_) | OidcAdminConfigError::Validation(_) => S3ErrorCode::InvalidRequest,
OidcAdminConfigError::StorageUnavailable
| OidcAdminConfigError::Load(_)
| OidcAdminConfigError::Save(_)
| OidcAdminConfigError::Task(_) => S3ErrorCode::InternalError,
};
S3Error::with_message(code, error.to_string())
}
/// Validate that a provider ID contains only safe characters (alphanumeric, underscore, hyphen).
fn is_valid_provider_id(id: &str) -> bool {
!id.is_empty() && id.chars().all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-')
@@ -198,104 +197,6 @@ struct OidcValidationResponse {
token_endpoint: Option<String>,
}
#[derive(Debug, Deserialize)]
#[serde(default, deny_unknown_fields)]
struct OidcConfigUpsertRequest {
enabled: bool,
display_name: String,
config_url: String,
issuer: Option<String>,
client_id: String,
client_secret: Option<String>,
scopes: Vec<String>,
other_audiences: Vec<String>,
redirect_uri: Option<String>,
redirect_uri_dynamic: bool,
claim_name: String,
claim_prefix: String,
role_policy: String,
groups_claim: String,
roles_claim: String,
email_claim: String,
username_claim: String,
hide_from_ui: bool,
}
impl Default for OidcConfigUpsertRequest {
fn default() -> Self {
Self {
enabled: true,
display_name: String::new(),
config_url: String::new(),
issuer: None,
client_id: String::new(),
client_secret: None,
scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(),
other_audiences: Vec::new(),
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: false,
}
}
}
#[derive(Debug, Deserialize)]
#[serde(default, deny_unknown_fields)]
struct OidcConfigValidateRequest {
provider_id: String,
enabled: bool,
display_name: String,
config_url: String,
issuer: Option<String>,
client_id: String,
client_secret: Option<String>,
scopes: Vec<String>,
other_audiences: Vec<String>,
redirect_uri: Option<String>,
redirect_uri_dynamic: bool,
claim_name: String,
claim_prefix: String,
role_policy: String,
groups_claim: String,
roles_claim: String,
email_claim: String,
username_claim: String,
hide_from_ui: bool,
}
impl Default for OidcConfigValidateRequest {
fn default() -> Self {
Self {
provider_id: "default".to_string(),
enabled: true,
display_name: String::new(),
config_url: String::new(),
issuer: None,
client_id: String::new(),
client_secret: None,
scopes: OIDC_DEFAULT_SCOPES.split(',').map(ToString::to_string).collect(),
other_audiences: Vec::new(),
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: false,
}
}
}
/// Handler: GET /rustfs/admin/v3/oidc/providers
/// Returns list of configured OIDC providers for the login page.
pub struct ListOidcProvidersHandler {}
@@ -323,9 +224,9 @@ impl Operation for GetOidcConfigHandler {
async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
authorize_oidc_config_request(&req, AdminAction::ServerInfoAdminAction).await?;
let config = load_server_config_from_store().await?;
let restart_required = oidc_restart_required(&config);
let providers = rustfs_iam::oidc::load_oidc_config_snapshot(Some(&config))
let config = oidc_config::list_config().await.map_err(config_service_error)?;
let providers = config
.snapshot
.into_providers()
.into_iter()
.map(|provider| OidcConfigView {
@@ -357,7 +258,7 @@ impl Operation for GetOidcConfigHandler {
StatusCode::OK,
&OidcConfigListResponse {
providers,
restart_required,
restart_required: config.restart_required,
},
)
}
@@ -373,22 +274,13 @@ impl Operation for PutOidcConfigHandler {
let provider_id = params
.get("provider_id")
.ok_or_else(|| s3_error!(InvalidRequest, "missing provider_id"))?;
if !is_valid_provider_id(provider_id) {
return Err(s3_error!(InvalidRequest, "invalid provider_id"));
}
if is_env_managed_provider(provider_id) {
return Err(s3_error!(AccessDenied, "provider is managed by environment variables"));
}
oidc_config::ensure_mutable_provider_id(provider_id).map_err(config_service_error)?;
let provider_id = provider_id.to_owned();
let request: OidcConfigUpsertRequest = parse_json_body(&mut req).await?;
update_oidc_server_config(move |config| {
let existing_secret = persisted_provider_secret(config, &provider_id);
let provider_config = build_provider_config_from_upsert(&provider_id, request, existing_secret)?;
upsert_persisted_provider_config(config, &provider_config);
Ok(())
})
.await?;
let request: OidcProviderConfigInput = parse_json_body(&mut req).await?;
oidc_config::upsert_config(provider_id, request)
.await
.map_err(config_service_error)?;
json_response(
StatusCode::OK,
@@ -411,19 +303,9 @@ impl Operation for DeleteOidcConfigHandler {
let provider_id = params
.get("provider_id")
.ok_or_else(|| s3_error!(InvalidRequest, "missing provider_id"))?;
if !is_valid_provider_id(provider_id) {
return Err(s3_error!(InvalidRequest, "invalid provider_id"));
}
if is_env_managed_provider(provider_id) {
return Err(s3_error!(AccessDenied, "provider is managed by environment variables"));
}
oidc_config::ensure_mutable_provider_id(provider_id).map_err(config_service_error)?;
let provider_id = provider_id.to_owned();
update_oidc_server_config(move |config| {
delete_persisted_provider_config(config, &provider_id)?;
Ok(())
})
.await?;
oidc_config::delete_config(provider_id).await.map_err(config_service_error)?;
json_response(
StatusCode::OK,
@@ -443,22 +325,8 @@ impl Operation for ValidateOidcConfigHandler {
async fn call(&self, mut req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
authorize_oidc_config_request(&req, AdminAction::ServerInfoAdminAction).await?;
let request: OidcConfigValidateRequest = parse_json_body(&mut req).await?;
let provider_id = if request.provider_id.trim().is_empty() {
"default".to_string()
} else {
request.provider_id.trim().to_string()
};
let provider_config = build_provider_config_from_validate(request, &provider_id)?;
let oidc_extra_root_ca = crate::startup_auth::current_oidc_extra_root_ca_material()
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?;
let validation = rustfs_iam::oidc::validate_oidc_provider_config_with_extra_root_ca(
&provider_config,
oidc_extra_root_ca.root_ca_pem.as_deref(),
)
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("validation failed: {e}")))?;
let request: OidcProviderValidationInput = parse_json_body(&mut req).await?;
let validation = oidc_config::validate_config(request).await.map_err(config_service_error)?;
json_response(
StatusCode::OK,
@@ -877,316 +745,6 @@ async fn parse_json_body<T: DeserializeOwned>(req: &mut S3Request<Body>) -> S3Re
serde_json::from_slice(&body).map_err(|e| s3_error!(InvalidRequest, "invalid JSON: {}", e))
}
async fn load_server_config_from_store() -> S3Result<ServerConfig> {
let store = oidc_config_store()?;
read_admin_config_without_migrate(store)
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to load server config: {e}")))
}
fn oidc_config_store() -> S3Result<std::sync::Arc<crate::admin::storage_api::runtime::ECStore>> {
let context = current_app_context();
current_object_store_handle_for_context(context.as_deref())
.ok_or_else(|| s3_error!(InternalError, "storage layer not initialized"))
}
async fn update_oidc_server_config<F>(modifier: F) -> S3Result<()>
where
F: FnOnce(&mut ServerConfig) -> S3Result<()> + Send + 'static,
{
let store = oidc_config_store()?;
supervise_admin_mutation("OIDC config update", async move {
let snapshot = read_admin_server_config_snapshot(store.clone())
.await
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to load server config: {e}")))?;
let mut config = snapshot.config.clone();
modifier(&mut config)?;
save_admin_server_config_snapshot(store, &config, &snapshot)
.await
.map(|_| ())
.map_err(|e| S3Error::with_message(S3ErrorCode::InternalError, format!("failed to save server config: {e}")))
})
.await
}
fn is_env_managed_provider(provider_id: &str) -> bool {
rustfs_iam::oidc::load_oidc_provider_configs_from_env()
.iter()
.any(|config| config.id == provider_id)
}
fn provider_instance_key(provider_id: &str) -> String {
if provider_id == "default" {
DEFAULT_DELIMITER.to_string()
} else {
provider_id.to_string()
}
}
fn oidc_restart_required(config: &ServerConfig) -> bool {
let context = current_app_context();
let active_config = current_server_config_for_context(context.as_deref());
oidc_restart_required_from_active_config(config, active_config.as_ref())
}
fn oidc_restart_required_from_active_config(config: &ServerConfig, active_config: Option<&ServerConfig>) -> bool {
rustfs_iam::oidc::load_oidc_config_snapshot(Some(config)) != rustfs_iam::oidc::load_oidc_config_snapshot(active_config)
}
fn default_oidc_kvs() -> s3s::S3Result<rustfs_config::server_config::KVS> {
ServerConfig::new()
.get_value(IDENTITY_OPENID_SUB_SYS, DEFAULT_DELIMITER)
.ok_or_else(|| s3_error!(InternalError, "default OIDC configuration missing"))
}
fn set_kvs_value(kvs: &mut rustfs_config::server_config::KVS, key: &str, value: String) {
if let Some(existing) = kvs.0.iter_mut().find(|kv| kv.key == key) {
existing.value = value;
return;
}
kvs.insert(key.to_string(), value);
}
fn normalize_scopes(scopes: &[String]) -> Vec<String> {
scopes
.iter()
.map(|scope| scope.trim().to_string())
.filter(|scope| !scope.is_empty())
.collect()
}
fn normalize_optional(value: Option<String>) -> Option<String> {
value.map(|v| v.trim().to_string()).filter(|v| !v.is_empty())
}
fn validate_absolute_http_url(value: &str, field_name: &str) -> S3Result<()> {
let parsed = Url::parse(value).map_err(|_| s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name))?;
if !is_valid_scheme(parsed.scheme()) || parsed.host_str().is_none() {
return Err(s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name));
}
validate_outbound_url(&parsed).map_err(|err| s3_error!(InvalidRequest, "{} is not allowed: {}", field_name, err))?;
Ok(())
}
fn validate_absolute_http_url_without_outbound_check(value: &str, field_name: &str) -> S3Result<()> {
let parsed = Url::parse(value).map_err(|_| s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name))?;
if !is_valid_scheme(parsed.scheme()) || parsed.host_str().is_none() {
return Err(s3_error!(InvalidRequest, "{} must be an absolute http/https URL", field_name));
}
Ok(())
}
fn validate_provider_config_fields(config: &rustfs_iam::oidc::OidcProviderConfig) -> S3Result<()> {
if !is_valid_provider_id(&config.id) {
return Err(s3_error!(InvalidRequest, "invalid provider_id"));
}
if config.config_url.trim().is_empty() {
return Err(s3_error!(InvalidRequest, "config_url is required"));
}
validate_absolute_http_url(&config.config_url, "config_url")?;
if let Some(issuer) = config.issuer.as_deref() {
validate_absolute_http_url_without_outbound_check(issuer, "issuer")?;
}
if config.client_id.trim().is_empty() {
return Err(s3_error!(InvalidRequest, "client_id is required"));
}
if !config.redirect_uri_dynamic {
let redirect_uri = config
.redirect_uri
.as_deref()
.ok_or_else(|| s3_error!(InvalidRequest, "redirect_uri is required when redirect_uri_dynamic is off"))?;
validate_absolute_http_url(redirect_uri, "redirect_uri")?;
} else if let Some(redirect_uri) = config.redirect_uri.as_deref() {
validate_absolute_http_url(redirect_uri, "redirect_uri")?;
}
if !config.scopes.iter().any(|scope| scope == "openid") {
return Err(s3_error!(InvalidRequest, "scopes must include openid"));
}
Ok(())
}
fn or_default(value: &str, default: &str) -> String {
if value.trim().is_empty() {
default.to_string()
} else {
value.trim().to_string()
}
}
/// Normalize an `OidcProviderConfig` by trimming strings and applying defaults.
fn normalize_provider_config(mut config: rustfs_iam::oidc::OidcProviderConfig) -> rustfs_iam::oidc::OidcProviderConfig {
config.config_url = config.config_url.trim().to_string();
config.issuer = normalize_optional(config.issuer);
config.client_id = config.client_id.trim().to_string();
config.scopes = normalize_scopes(&config.scopes);
config.redirect_uri = normalize_optional(config.redirect_uri);
config.claim_name = or_default(&config.claim_name, OIDC_DEFAULT_CLAIM_NAME);
config.claim_prefix = config.claim_prefix.trim().to_string();
config.role_policy = config.role_policy.trim().to_string();
config.display_name = or_default(&config.display_name, &config.id);
config.groups_claim = or_default(&config.groups_claim, OIDC_DEFAULT_GROUPS_CLAIM);
config.roles_claim = or_default(&config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM);
config.email_claim = or_default(&config.email_claim, OIDC_DEFAULT_EMAIL_CLAIM);
config.username_claim = or_default(&config.username_claim, OIDC_DEFAULT_USERNAME_CLAIM);
config
}
fn build_provider_config_from_upsert(
provider_id: &str,
request: OidcConfigUpsertRequest,
existing_secret: Option<String>,
) -> S3Result<rustfs_iam::oidc::OidcProviderConfig> {
let client_secret = match request.client_secret {
Some(value) if !value.trim().is_empty() => Some(value),
_ => existing_secret.filter(|value| !value.trim().is_empty()),
};
let config = normalize_provider_config(rustfs_iam::oidc::OidcProviderConfig {
id: provider_id.to_string(),
enabled: request.enabled,
config_url: request.config_url,
issuer: request.issuer,
client_id: request.client_id,
client_secret,
scopes: request.scopes,
other_audiences: request.other_audiences,
redirect_uri: request.redirect_uri,
redirect_uri_dynamic: request.redirect_uri_dynamic,
claim_name: request.claim_name,
claim_prefix: request.claim_prefix,
role_policy: request.role_policy,
display_name: request.display_name,
groups_claim: request.groups_claim,
roles_claim: request.roles_claim,
email_claim: request.email_claim,
username_claim: request.username_claim,
hide_from_ui: request.hide_from_ui,
});
validate_provider_config_fields(&config)?;
Ok(config)
}
fn build_provider_config_from_validate(
request: OidcConfigValidateRequest,
provider_id: &str,
) -> S3Result<rustfs_iam::oidc::OidcProviderConfig> {
let config = normalize_provider_config(rustfs_iam::oidc::OidcProviderConfig {
id: provider_id.to_string(),
enabled: request.enabled,
config_url: request.config_url,
issuer: request.issuer,
client_id: request.client_id,
client_secret: request.client_secret.filter(|value| !value.trim().is_empty()),
scopes: request.scopes,
other_audiences: request.other_audiences,
redirect_uri: request.redirect_uri,
redirect_uri_dynamic: request.redirect_uri_dynamic,
claim_name: request.claim_name,
claim_prefix: request.claim_prefix,
role_policy: request.role_policy,
display_name: request.display_name,
groups_claim: request.groups_claim,
roles_claim: request.roles_claim,
email_claim: request.email_claim,
username_claim: request.username_claim,
hide_from_ui: request.hide_from_ui,
});
validate_provider_config_fields(&config)?;
Ok(config)
}
fn persisted_provider_secret(config: &ServerConfig, provider_id: &str) -> Option<String> {
config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|subsystem| subsystem.get(&provider_instance_key(provider_id)))
.and_then(|kvs| kvs.lookup(OIDC_CLIENT_SECRET))
.filter(|value| !value.trim().is_empty())
}
fn upsert_persisted_provider_config(config: &mut ServerConfig, provider_config: &rustfs_iam::oidc::OidcProviderConfig) {
let instance_key = provider_instance_key(&provider_config.id);
let mut kvs = default_oidc_kvs().unwrap_or_default();
set_kvs_value(
&mut kvs,
ENABLE_KEY,
if provider_config.enabled {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
set_kvs_value(&mut kvs, OIDC_CONFIG_URL, provider_config.config_url.clone());
set_kvs_value(&mut kvs, OIDC_ISSUER, provider_config.issuer.clone().unwrap_or_default());
set_kvs_value(&mut kvs, OIDC_CLIENT_ID, provider_config.client_id.clone());
set_kvs_value(&mut kvs, OIDC_CLIENT_SECRET, provider_config.client_secret.clone().unwrap_or_default());
set_kvs_value(&mut kvs, OIDC_SCOPES, provider_config.scopes.join(","));
set_kvs_value(&mut kvs, OIDC_OTHER_AUDIENCES, provider_config.other_audiences.join(","));
set_kvs_value(&mut kvs, OIDC_REDIRECT_URI, provider_config.redirect_uri.clone().unwrap_or_default());
set_kvs_value(
&mut kvs,
OIDC_REDIRECT_URI_DYNAMIC,
if provider_config.redirect_uri_dynamic {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
set_kvs_value(&mut kvs, OIDC_CLAIM_NAME, provider_config.claim_name.clone());
set_kvs_value(&mut kvs, OIDC_CLAIM_PREFIX, provider_config.claim_prefix.clone());
set_kvs_value(&mut kvs, OIDC_ROLE_POLICY, provider_config.role_policy.clone());
set_kvs_value(&mut kvs, OIDC_DISPLAY_NAME, provider_config.display_name.clone());
set_kvs_value(&mut kvs, OIDC_GROUPS_CLAIM, provider_config.groups_claim.clone());
set_kvs_value(&mut kvs, OIDC_ROLES_CLAIM, provider_config.roles_claim.clone());
set_kvs_value(&mut kvs, OIDC_EMAIL_CLAIM, provider_config.email_claim.clone());
set_kvs_value(&mut kvs, OIDC_USERNAME_CLAIM, provider_config.username_claim.clone());
set_kvs_value(
&mut kvs,
OIDC_HIDE_FROM_UI,
if provider_config.hide_from_ui {
EnableState::On.to_string()
} else {
EnableState::Off.to_string()
},
);
config
.0
.entry(IDENTITY_OPENID_SUB_SYS.to_string())
.or_default()
.insert(instance_key, kvs);
}
fn delete_persisted_provider_config(config: &mut ServerConfig, provider_id: &str) -> S3Result<()> {
let Some(subsystem) = config.0.get_mut(IDENTITY_OPENID_SUB_SYS) else {
return Err(s3_error!(InvalidRequest, "provider not found"));
};
if subsystem.remove(&provider_instance_key(provider_id)).is_none() {
return Err(s3_error!(InvalidRequest, "provider not found"));
}
if subsystem.is_empty() {
config.0.remove(IDENTITY_OPENID_SUB_SYS);
}
Ok(())
}
fn extract_request_scheme(req: &S3Request<Body>) -> S3Result<String> {
let raw_scheme = req
.headers
@@ -1739,191 +1297,36 @@ mod tests {
}
#[test]
fn test_validate_absolute_http_url_rejects_loopback_targets() {
let err = validate_absolute_http_url("https://127.0.0.1/.well-known/openid-configuration", "config_url")
.expect_err("loopback config URL should be rejected");
assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
assert!(err.message().unwrap_or_default().contains("not allowed"));
}
fn oidc_config_errors_keep_admin_codes() {
let error = config_service_error(oidc_config::OidcAdminConfigError::Config(
rustfs_iam::oidc::OidcConfigError::EnvironmentManaged,
));
assert_eq!(error.code(), &S3ErrorCode::AccessDenied);
assert_eq!(error.message(), Some("provider is managed by environment variables"));
#[test]
fn test_provider_instance_key() {
assert_eq!(provider_instance_key("default"), "_");
assert_eq!(provider_instance_key("okta"), "okta");
}
let error = config_service_error(oidc_config::OidcAdminConfigError::Config(
rustfs_iam::oidc::OidcConfigError::ProviderNotFound,
));
assert_eq!(error.code(), &S3ErrorCode::InvalidRequest);
assert_eq!(error.message(), Some("provider not found"));
#[test]
fn test_build_provider_config_requires_openid_scope() {
let req = OidcConfigUpsertRequest {
scopes: vec!["profile".to_string()],
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client-id".to_string(),
..Default::default()
};
let error = config_service_error(oidc_config::OidcAdminConfigError::Task("cancelled"));
assert_eq!(error.code(), &S3ErrorCode::InternalError);
assert_eq!(error.message(), Some("OIDC config update task cancelled"));
assert!(build_provider_config_from_upsert("default", req, None).is_err());
}
#[test]
fn test_build_provider_config_preserves_existing_secret_when_request_is_empty() {
let req = OidcConfigUpsertRequest {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client-id".to_string(),
client_secret: Some("".to_string()),
..Default::default()
};
let config =
build_provider_config_from_upsert("default", req, Some("existing-secret".to_string())).expect("config should build");
assert_eq!(config.client_secret.as_deref(), Some("existing-secret"));
assert_eq!(config.roles_claim, OIDC_DEFAULT_ROLES_CLAIM);
}
#[test]
fn test_oidc_config_upsert_request_rejects_unknown_fields() {
let err = serde_json::from_str::<OidcConfigUpsertRequest>(
r#"{"config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#,
let error = rustfs_iam::oidc::build_upsert_provider_config(
"default",
rustfs_iam::oidc::OidcProviderConfigInput {
config_url: "https://127.0.0.1/.well-known/openid-configuration".to_string(),
client_id: "client".to_string(),
..Default::default()
},
None,
)
.expect_err("unknown upsert field should fail");
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn test_oidc_config_validate_request_rejects_unknown_fields() {
let err = serde_json::from_str::<OidcConfigValidateRequest>(
r#"{"provider_id":"default","config_url":"https://example.com/.well-known/openid-configuration","client_id":"client","unexpected_field":true}"#,
)
.expect_err("unknown validate field should fail");
assert!(err.to_string().contains("unknown field"));
}
#[test]
fn test_build_provider_config_uses_custom_roles_claim() {
let req = OidcConfigUpsertRequest {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "client-id".to_string(),
roles_claim: "app_roles".to_string(),
..Default::default()
};
let config = build_provider_config_from_upsert("default", req, None).expect("config should build");
assert_eq!(config.roles_claim, "app_roles");
}
#[test]
fn test_oidc_restart_required_detects_persisted_changes() {
let active_config = ServerConfig::new();
let mut persisted_config = ServerConfig::new();
let provider_config = rustfs_iam::oidc::OidcProviderConfig {
id: "default".to_string(),
enabled: true,
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
issuer: None,
client_id: "console".to_string(),
client_secret: Some("secret".to_string()),
scopes: vec!["openid".to_string(), "profile".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: OIDC_DEFAULT_CLAIM_NAME.to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
display_name: "default".to_string(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: false,
};
upsert_persisted_provider_config(&mut persisted_config, &provider_config);
assert!(oidc_restart_required_from_active_config(&persisted_config, Some(&active_config)));
assert!(!oidc_restart_required_from_active_config(&persisted_config, Some(&persisted_config)));
}
#[test]
fn test_upsert_persists_hide_from_ui_on() {
let mut config = ServerConfig::new();
let mut provider_config = rustfs_iam::oidc::OidcProviderConfig {
id: "kubernetes".to_string(),
enabled: true,
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
issuer: None,
client_id: "test".to_string(),
client_secret: None,
scopes: vec!["openid".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: "sub".to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
display_name: "Kubernetes".to_string(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: true,
};
upsert_persisted_provider_config(&mut config, &provider_config);
let kvs = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|m| m.get("kubernetes"))
.expect("provider KVS should exist");
assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::On.to_string());
// Flip to false and verify
provider_config.hide_from_ui = false;
upsert_persisted_provider_config(&mut config, &provider_config);
let kvs = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|m| m.get("kubernetes"))
.expect("provider KVS should exist");
assert_eq!(kvs.get(OIDC_HIDE_FROM_UI), EnableState::Off.to_string());
}
#[test]
fn test_upsert_persists_issuer() {
let mut config = ServerConfig::new();
let provider_config = rustfs_iam::oidc::OidcProviderConfig {
id: "kubernetes".to_string(),
enabled: true,
config_url: "http://keycloak.ns.svc.cluster.local:8080/realms/app/.well-known/openid-configuration".to_string(),
issuer: Some("https://app.local/realms/app".to_string()),
client_id: "test".to_string(),
client_secret: None,
scopes: vec!["openid".to_string()],
other_audiences: vec![],
redirect_uri: None,
redirect_uri_dynamic: true,
claim_name: "sub".to_string(),
claim_prefix: String::new(),
role_policy: String::new(),
display_name: "Kubernetes".to_string(),
groups_claim: OIDC_DEFAULT_GROUPS_CLAIM.to_string(),
roles_claim: OIDC_DEFAULT_ROLES_CLAIM.to_string(),
email_claim: OIDC_DEFAULT_EMAIL_CLAIM.to_string(),
username_claim: OIDC_DEFAULT_USERNAME_CLAIM.to_string(),
hide_from_ui: false,
};
upsert_persisted_provider_config(&mut config, &provider_config);
let kvs = config
.0
.get(IDENTITY_OPENID_SUB_SYS)
.and_then(|m| m.get("kubernetes"))
.expect("provider KVS should exist");
assert_eq!(kvs.get(OIDC_ISSUER), "https://app.local/realms/app");
.expect_err("loopback config URL should be rejected");
let error = config_service_error(oidc_config::OidcAdminConfigError::Config(error));
assert_eq!(error.code(), &S3ErrorCode::InvalidRequest);
assert!(error.message().unwrap_or_default().contains("not allowed"));
}
/// The OIDC config gate now authorizes through the shared admin gate, which
rustfs/src/admin/handlers/site_replication.rs
+21 -163
View File
@@ -15,7 +15,7 @@
use crate::admin::auth::authorize_admin_request;
use crate::admin::router::{AdminOperation, Operation, S3Router};
use crate::admin::runtime_sources::{
current_deployment_id, current_federated_identity_runtime, current_iam_handle, current_object_store_handle, current_region,
current_deployment_id, current_iam_handle, current_object_store_handle, current_oidc_config_query, current_region,
current_replication_pool_handle, current_replication_stats_handle, current_server_config, current_token_signing_key,
object_store_from_req,
};
@@ -2063,63 +2063,32 @@ fn open_id_settings(providers: Vec<(String, OpenIDProviderSettings)>, region: St
settings
}
fn open_id_settings_from_snapshot(snapshot: &rustfs_iam::oidc::OidcConfigSnapshot, region: String) -> OpenIDSettings {
fn open_id_settings_from_snapshot(snapshot: &rustfs_iam::oidc::OidcSiteReplicationSnapshot, region: String) -> OpenIDSettings {
let providers = snapshot
.providers()
.iter()
.map(|provider| {
let config = &provider.config;
(config.id.clone(), open_id_provider_settings(config))
(
provider.provider_id.clone(),
OpenIDProviderSettings {
claim_name: provider.claim_name.clone(),
claim_userinfo_enabled: false,
role_policy: provider.role_policy.clone(),
client_id: provider.client_id.clone(),
hashed_client_secret: provider.hashed_client_secret.clone(),
},
)
})
.collect();
open_id_settings(providers, region)
}
fn open_id_provider_settings(config: &rustfs_iam::oidc::OidcProviderConfig) -> OpenIDProviderSettings {
OpenIDProviderSettings {
claim_name: config.claim_name.clone(),
claim_userinfo_enabled: false,
role_policy: config.role_policy.clone(),
client_id: config.client_id.clone(),
hashed_client_secret: hash_client_secret(config.client_secret.as_deref()),
}
}
fn open_id_settings_from_service(service: &rustfs_iam::federation::FederatedIdentityService, region: String) -> OpenIDSettings {
let providers = service
.list_providers()
.into_iter()
.filter_map(|provider| {
let config = service.get_provider_config(&provider.provider_id)?;
Some((provider.provider_id, open_id_provider_settings(config)))
})
.collect();
open_id_settings(providers, region)
}
fn open_id_settings_from_runtime(
oidc_config_query: Option<&dyn rustfs_iam::federation::oidc::OidcConfigQuery>,
service: Option<&rustfs_iam::federation::FederatedIdentityService>,
region: String,
) -> OpenIDSettings {
if let Some(oidc_config_query) = oidc_config_query {
return open_id_settings_from_snapshot(&oidc_config_query.config_snapshot(), region);
}
service
.map(|service| open_id_settings_from_service(service, region))
.unwrap_or_default()
}
fn local_idp_settings() -> IDPSettings {
let mut settings = IDPSettings::default();
let region = current_region().map(|region| region.to_string()).unwrap_or_default();
settings.open_id = current_federated_identity_runtime()
.map(|(service, oidc_config_query)| {
open_id_settings_from_runtime(oidc_config_query.as_deref(), Some(service.as_ref()), region)
})
settings.open_id = current_oidc_config_query()
.map(|query| open_id_settings_from_snapshot(&query.site_replication_snapshot(), region))
.unwrap_or_default();
let (ldap, ldap_configs) = load_ldap_idp_settings();
@@ -11444,82 +11413,6 @@ mod tests {
}
}
struct LegacyOidcProvider {
configs: Vec<rustfs_iam::oidc::OidcProviderConfig>,
}
struct SnapshotOidcConfigQuery {
snapshot: rustfs_iam::oidc::OidcConfigSnapshot,
}
impl rustfs_iam::federation::oidc::OidcConfigQuery for SnapshotOidcConfigQuery {
fn config_snapshot(&self) -> rustfs_iam::oidc::OidcConfigSnapshot {
self.snapshot.clone()
}
}
#[async_trait::async_trait]
impl rustfs_iam::federation::FederatedIdentityProvider for LegacyOidcProvider {
fn has_providers(&self) -> bool {
!self.configs.is_empty()
}
fn list_providers(&self) -> Vec<rustfs_iam::federation::FederatedProviderView> {
self.configs
.iter()
.map(|config| rustfs_iam::federation::FederatedProviderView {
provider_id: config.id.clone(),
display_name: config.display_name.clone(),
})
.collect()
}
fn list_visible_providers(&self) -> Vec<rustfs_iam::federation::FederatedProviderView> {
self.list_providers()
}
fn provider_config(&self, provider_id: &str) -> Option<&rustfs_iam::oidc::OidcProviderConfig> {
self.configs.iter().find(|config| config.id == provider_id)
}
async fn authorize_url(
&self,
_provider_id: &str,
_redirect_uri: &str,
_redirect_after: Option<String>,
) -> rustfs_iam::federation::Result<String> {
unreachable!("site replication compatibility test does not authorize")
}
async fn exchange_code(
&self,
_state: &str,
_code: &str,
_redirect_uri: &str,
) -> rustfs_iam::federation::Result<rustfs_iam::federation::FederatedCodeExchange> {
unreachable!("site replication compatibility test does not exchange codes")
}
async fn verify_web_identity_token(
&self,
_jwt: &str,
) -> rustfs_iam::federation::Result<rustfs_iam::federation::FederatedAuthorization> {
unreachable!("site replication compatibility test does not verify tokens")
}
async fn create_logout_token(&self, _provider_id: &str, _id_token: &str) -> rustfs_iam::federation::Result<String> {
unreachable!("site replication compatibility test does not create logout tokens")
}
async fn build_logout_url(
&self,
_logout_token: &str,
_post_logout_redirect_uri: &str,
) -> rustfs_iam::federation::Result<Option<String>> {
unreachable!("site replication compatibility test does not build logout URLs")
}
}
fn ldap_settings_kvs() -> rustfs_config::server_config::KVS {
rustfs_config::server_config::KVS(vec![
rustfs_config::server_config::KV {
@@ -11652,16 +11545,18 @@ mod tests {
#[test]
fn oidc_config_snapshot_preserves_site_replication_settings() {
let snapshot = rustfs_iam::oidc::OidcConfigSnapshot::new(vec![
oidc_snapshot_provider("corp", "client-b"),
oidc_snapshot_provider("default", "client-a"),
]);
let snapshot = rustfs_iam::oidc::OidcSiteReplicationSnapshot::from_config_snapshot(
&rustfs_iam::oidc::OidcConfigSnapshot::new(vec![
oidc_snapshot_provider("corp", "client-b"),
oidc_snapshot_provider("default", "client-a"),
]),
);
assert_eq!(
snapshot
.providers()
.iter()
.map(|provider| provider.config.id.as_str())
.map(|provider| provider.provider_id.as_str())
.collect::<Vec<_>>(),
["default", "corp"]
);
@@ -11692,43 +11587,6 @@ mod tests {
);
}
#[test]
fn service_only_oidc_runtime_preserves_site_replication_settings() {
let default = oidc_snapshot_provider("default", "client-a");
let corp = oidc_snapshot_provider("corp", "client-b");
let expected = open_id_settings_from_snapshot(
&rustfs_iam::oidc::OidcConfigSnapshot::new(vec![corp.clone(), default.clone()]),
"eu-site-1".to_string(),
);
let service = rustfs_iam::federation::FederatedIdentityService::new(
rustfs_iam::federation::FederatedIdentityRegistry::new(Arc::new(LegacyOidcProvider {
configs: vec![corp.config, default.config],
})),
);
assert_eq!(
serde_json::to_value(open_id_settings_from_runtime(None, Some(&service), "eu-site-1".to_string(),))
.expect("serialize legacy OpenID settings"),
serde_json::to_value(expected).expect("serialize snapshot OpenID settings"),
);
}
#[test]
fn oidc_query_takes_precedence_over_service_only_compatibility() {
let query = SnapshotOidcConfigQuery {
snapshot: rustfs_iam::oidc::OidcConfigSnapshot::new(vec![oidc_snapshot_provider("default", "query-client")]),
};
let service = rustfs_iam::federation::FederatedIdentityService::new(
rustfs_iam::federation::FederatedIdentityRegistry::new(Arc::new(LegacyOidcProvider {
configs: vec![oidc_snapshot_provider("default", "service-client").config],
})),
);
let settings = open_id_settings_from_runtime(Some(&query), Some(&service), "eu-site-1".to_string());
assert_eq!(settings.claim_provider.client_id, "query-client");
}
// A whole provider object present on one side only must not spill its
// credential-derived leaves into the rejection message.
#[test]
rustfs/src/admin/runtime_sources.rs
+5 -6
View File
@@ -23,12 +23,11 @@ use crate::app::object_usecase::DefaultObjectUsecase;
use crate::runtime_sources as root_runtime_sources;
pub(crate) use crate::runtime_sources::{
AppContext, ServerContextSlot, current_action_credentials, current_boot_time, current_bucket_metadata_handle,
current_bucket_monitor_handle, current_deployment_id, current_endpoints_handle, current_federated_identity_runtime,
current_federated_identity_service, current_iam_handle, current_kms_runtime_service_manager,
current_notification_system_for_context, current_object_data_cache_handle_for_context,
current_object_store_handle_for_context, current_ready_iam_handle, current_region, current_replication_pool_handle,
current_replication_stats_handle, current_replication_stats_handle_for_context, current_server_config_for_context,
current_token_signing_key,
current_bucket_monitor_handle, current_deployment_id, current_endpoints_handle, current_federated_identity_service,
current_iam_handle, current_kms_runtime_service_manager, current_notification_system_for_context,
current_object_data_cache_handle_for_context, current_object_store_handle_for_context, current_oidc_config_query,
current_ready_iam_handle, current_region, current_replication_pool_handle, current_replication_stats_handle,
current_replication_stats_handle_for_context, current_server_config_for_context, current_token_signing_key,
};
#[cfg(test)]
pub(crate) use crate::runtime_sources::{
rustfs/src/admin/service/mod.rs
+1
View File
@@ -15,5 +15,6 @@
pub(crate) mod caller_identity;
pub mod config;
pub(crate) mod federated_identity;
pub(crate) mod oidc_config;
pub(crate) mod session_policy;
pub mod site_replication;
rustfs/src/admin/service/oidc_config.rs
+158
View File
@@ -0,0 +1,158 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::admin::runtime_sources::{
current_app_context, current_object_store_handle_for_context, current_server_config_for_context,
};
use crate::admin::storage_api::config::{
read_admin_config_without_migrate, read_admin_server_config_snapshot, save_admin_server_config_snapshot,
};
use crate::admin::storage_api::error::Error as StorageError;
use rustfs_config::server_config::Config as ServerConfig;
use rustfs_iam::oidc::{
OidcConfigError, OidcConfigSnapshot, OidcProviderConfigInput, OidcProviderValidationInput, OidcProviderValidationResult,
build_upsert_provider_config, build_validation_provider_config, delete_persisted_provider_config, load_oidc_config_snapshot,
persisted_provider_secret, upsert_persisted_provider_config, validate_mutable_provider_id,
validate_oidc_provider_config_with_extra_root_ca,
};
#[derive(Debug, thiserror::Error)]
pub(crate) enum OidcAdminConfigError {
#[error("storage layer not initialized")]
StorageUnavailable,
#[error("failed to load server config: {0}")]
Load(#[source] StorageError),
#[error("failed to save server config: {0}")]
Save(#[source] StorageError),
#[error("validation failed: {0}")]
Validation(String),
#[error("OIDC config update task {0}")]
Task(&'static str),
#[error(transparent)]
Config(#[from] OidcConfigError),
}
type Result<T> = std::result::Result<T, OidcAdminConfigError>;
pub(crate) struct OidcConfigList {
pub snapshot: OidcConfigSnapshot,
pub restart_required: bool,
}
pub(crate) fn ensure_mutable_provider_id(provider_id: &str) -> Result<()> {
validate_mutable_provider_id(provider_id)?;
Ok(())
}
fn oidc_config_store() -> Result<std::sync::Arc<crate::admin::storage_api::runtime::ECStore>> {
let context = current_app_context();
current_object_store_handle_for_context(context.as_deref()).ok_or(OidcAdminConfigError::StorageUnavailable)
}
async fn load_server_config_from_store() -> Result<ServerConfig> {
let store = oidc_config_store()?;
read_admin_config_without_migrate(store)
.await
.map_err(OidcAdminConfigError::Load)
}
fn restart_required_from_active_config(config: &ServerConfig, active_config: Option<&ServerConfig>) -> bool {
load_oidc_config_snapshot(Some(config)) != load_oidc_config_snapshot(active_config)
}
pub(crate) async fn list_config() -> Result<OidcConfigList> {
let config = load_server_config_from_store().await?;
let context = current_app_context();
let active_config = current_server_config_for_context(context.as_deref());
Ok(OidcConfigList {
snapshot: load_oidc_config_snapshot(Some(&config)),
restart_required: restart_required_from_active_config(&config, active_config.as_ref()),
})
}
async fn update_server_config<F>(modifier: F) -> Result<()>
where
F: FnOnce(&mut ServerConfig) -> Result<()> + Send + 'static,
{
let store = oidc_config_store()?;
tokio::spawn(async move {
let snapshot = read_admin_server_config_snapshot(store.clone())
.await
.map_err(OidcAdminConfigError::Load)?;
let mut config = snapshot.config.clone();
modifier(&mut config)?;
save_admin_server_config_snapshot(store, &config, &snapshot)
.await
.map(|_| ())
.map_err(OidcAdminConfigError::Save)
})
.await
.map_err(|error| OidcAdminConfigError::Task(if error.is_cancelled() { "cancelled" } else { "panicked" }))?
}
pub(crate) async fn upsert_config(provider_id: String, input: OidcProviderConfigInput) -> Result<()> {
ensure_mutable_provider_id(&provider_id)?;
update_server_config(move |config| {
let existing_secret = persisted_provider_secret(config, &provider_id);
let provider = build_upsert_provider_config(&provider_id, input, existing_secret)?;
upsert_persisted_provider_config(config, &provider);
Ok(())
})
.await
}
pub(crate) async fn delete_config(provider_id: String) -> Result<()> {
ensure_mutable_provider_id(&provider_id)?;
update_server_config(move |config| {
delete_persisted_provider_config(config, &provider_id)?;
Ok(())
})
.await
}
pub(crate) async fn validate_config(input: OidcProviderValidationInput) -> Result<OidcProviderValidationResult> {
let provider = build_validation_provider_config(input)?;
let extra_root_ca = crate::startup_auth::current_oidc_extra_root_ca_material()
.await
.map_err(|error| OidcAdminConfigError::Validation(error.to_string()))?;
validate_oidc_provider_config_with_extra_root_ca(&provider, extra_root_ca.root_ca_pem.as_deref())
.await
.map_err(OidcAdminConfigError::Validation)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn restart_required_detects_persisted_changes() {
let active_config = ServerConfig::new();
let mut persisted_config = ServerConfig::new();
let provider = build_upsert_provider_config(
"default",
OidcProviderConfigInput {
config_url: "https://example.com/.well-known/openid-configuration".to_string(),
client_id: "console".to_string(),
client_secret: Some("secret".to_string()),
..Default::default()
},
None,
)
.expect("provider config should be valid");
upsert_persisted_provider_config(&mut persisted_config, &provider);
assert!(restart_required_from_active_config(&persisted_config, Some(&active_config)));
assert!(!restart_required_from_active_config(&persisted_config, Some(&persisted_config)));
}
}
rustfs/src/app/bucket_usecase.rs
+93 -2
View File
@@ -1086,7 +1086,11 @@ fn build_list_object_versions_metadata_output(
size: Some(object.size),
version_id: Some(version_id),
is_latest: Some(object.is_latest),
e_tag: object.etag.clone().map(|etag| to_s3s_etag(&etag)),
e_tag: object
.etag
.as_ref()
.filter(|_| rustfs_utils::http::etag_can_be_listed(&object.user_defined))
.map(|etag| to_s3s_etag(etag)),
storage_class: Some(ObjectVersionStorageClass::from(
object
.storage_class
@@ -1173,7 +1177,11 @@ fn build_list_objects_v2_metadata_output(
key: Some(encode_list_objects_v2_value(&object.name, encoding_type)),
last_modified: object.mod_time.map(Timestamp::from),
size: Some(object.get_actual_size_or_physical()),
e_tag: object.etag.clone().map(|etag| to_s3s_etag(&etag)),
e_tag: object
.etag
.as_ref()
.filter(|_| rustfs_utils::http::etag_can_be_listed(&object.user_defined))
.map(|etag| to_s3s_etag(etag)),
storage_class: Some(ObjectStorageClass::from(
object
.storage_class
@@ -4548,6 +4556,89 @@ mod tests {
assert_eq!(err.code(), &S3ErrorCode::InternalError);
}
#[test]
fn keyless_object_and_version_listings_do_not_expose_legacy_encrypted_fingerprints() {
let legacy_md5 = "fbfcbcc6d2f035411e6268c8b119593e";
let mut protected_metadata =
HashMap::from([("x-amz-server-side-encryption-customer-algorithm".to_owned(), "AES256".to_owned())]);
rustfs_utils::http::metadata_compat::insert_str(
&mut protected_metadata,
rustfs_utils::http::SUFFIX_OPAQUE_ENCRYPTED_ETAG,
"v1".to_owned(),
);
let objects = vec![
ObjectInfo {
name: "legacy-encrypted".to_owned(),
etag: Some(legacy_md5.to_owned()),
user_defined: Arc::new(HashMap::from([(
"x-amz-server-side-encryption-customer-algorithm".to_owned(),
"AES256".to_owned(),
)])),
..Default::default()
},
ObjectInfo {
name: "protected-encrypted".to_owned(),
etag: Some("0123456789abcdef0123456789abcdef".to_owned()),
user_defined: Arc::new(protected_metadata),
..Default::default()
},
ObjectInfo {
name: "plain".to_owned(),
etag: Some("9876543210abcdef9876543210abcdef".to_owned()),
..Default::default()
},
];
let listings = build_list_objects_v2_metadata_output(
ListObjectsV2Info {
objects: objects.clone(),
..Default::default()
},
"demo-bucket",
&ListObjectsV2Params {
prefix: String::new(),
max_keys: 1000,
delimiter: None,
response_start_after: None,
start_after_for_query: None,
response_continuation_token: None,
decoded_continuation_token: None,
},
None,
false,
&HashMap::new(),
);
assert!(listings.contents[0].object.e_tag.is_none());
assert!(listings.contents[1].object.e_tag.is_some());
assert!(listings.contents[2].object.e_tag.is_some());
let xml = String::from_utf8(serialize_config(&listings).expect("serialize objects")).expect("UTF-8 objects");
assert!(!xml.contains(legacy_md5));
let versions = build_list_object_versions_metadata_output(
ListObjectVersionsInfo {
objects,
..Default::default()
},
"demo-bucket",
&ListObjectVersionsParams {
prefix: String::new(),
max_keys: 1000,
delimiter: None,
key_marker: None,
version_id_marker: None,
},
None,
&HashMap::new(),
);
for (index, entry) in versions.entries.iter().enumerate() {
let ListObjectVersionMetadataEntry::Version(version, _) = entry else {
panic!("expected object version");
};
assert_eq!(version.e_tag.is_some(), index != 0);
}
let xml = String::from_utf8(serialize_config(&versions).expect("serialize versions")).expect("UTF-8 versions");
assert!(!xml.contains(legacy_md5));
}
#[test]
fn build_list_object_versions_metadata_output_maps_metadata_and_preserves_entry_order() {
use time::macros::datetime;
rustfs/src/app/context.rs
+8 -39
View File
@@ -42,9 +42,7 @@ use crate::config::RustFSBufferConfig;
use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials;
use rustfs_iam::{
error::Error as IamError,
federation::{FederatedIdentityService, oidc::OidcConfigQuery},
store::object::ObjectStore,
error::Error as IamError, federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore,
sys::IamSys,
};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
@@ -106,11 +104,6 @@ pub fn resolve_oidc_config_query() -> Option<Arc<dyn OidcConfigQuery>> {
resolve_oidc_config_query_with(get_global_app_context())
}
/// Resolve the paired federation service and OIDC configuration query from one runtime observation.
pub(crate) fn resolve_federated_identity_runtime() -> Option<FederatedIdentityRuntimeSnapshot> {
resolve_federated_identity_runtime_with(get_global_app_context())
}
/// Publish the initialized federation service and OIDC configuration query.
pub fn publish_federated_identity_runtime(
service: Arc<FederatedIdentityService>,
@@ -119,11 +112,6 @@ pub fn publish_federated_identity_runtime(
handles::publish_default_federated_identity_runtime(service, oidc_config_query)
}
/// Publish a service-only federation runtime for compatibility integrations.
pub fn publish_federated_identity_service(service: Arc<FederatedIdentityService>) -> bool {
handles::publish_default_federated_identity_service(service)
}
/// Resolve a ready IAM system handle using AppContext-first precedence.
pub fn resolve_ready_iam_handle() -> rustfs_iam::error::Result<Arc<IamSys<ObjectStore>>> {
if let Some(context) = get_global_app_context() {
@@ -363,6 +351,7 @@ fn resolve_oidc_config_query_with(context: Option<Arc<AppContext>>) -> Option<Ar
context.and_then(|context| context.federated_identity().oidc_config_query())
}
#[cfg(test)]
fn resolve_federated_identity_runtime_with(context: Option<Arc<AppContext>>) -> Option<FederatedIdentityRuntimeSnapshot> {
context.and_then(|context| context.federated_identity().runtime_snapshot())
}
@@ -556,11 +545,8 @@ mod tests {
use crate::config::{RustFSBufferConfig, WorkloadProfile};
use async_trait::async_trait;
use rustfs_iam::{
federation::{
FederatedIdentityRegistry, FederatedIdentityService,
oidc::{OidcConfigQuery, StandardOidcAdapter},
},
oidc::OidcSys,
federation::{CoreFederatedAuthorizationMapper, FederatedIdentityService},
oidc::{OidcConfigQuery, OidcSys, StandardOidcAdapter},
store::object::ObjectStore,
sys::IamSys,
};
@@ -620,25 +606,11 @@ mod tests {
}
impl FederatedIdentityInterface for TestFederatedIdentityInterface {
fn handle(&self) -> Option<Arc<FederatedIdentityService>> {
self.runtime
.read()
.ok()
.and_then(|runtime| runtime.as_ref().map(|runtime| Arc::clone(&runtime.service)))
}
fn oidc_config_query(&self) -> Option<Arc<dyn OidcConfigQuery>> {
self.runtime
.read()
.ok()
.and_then(|runtime| runtime.as_ref().map(|runtime| Arc::clone(&runtime.oidc_config_query)))
}
fn runtime_snapshot(&self) -> Option<FederatedIdentityRuntimeSnapshot> {
self.runtime.read().ok().and_then(|runtime| {
runtime
.as_ref()
.map(|runtime| (Arc::clone(&runtime.service), Some(Arc::clone(&runtime.oidc_config_query))))
.map(|runtime| (Arc::clone(&runtime.service), Arc::clone(&runtime.oidc_config_query)))
})
}
@@ -657,7 +629,8 @@ mod tests {
fn test_federated_identity_runtime(oidc: OidcSys) -> (Arc<FederatedIdentityService>, Arc<dyn OidcConfigQuery>) {
let adapter = Arc::new(StandardOidcAdapter::new(Arc::new(oidc)));
let oidc_config_query = adapter.clone();
let service = Arc::new(FederatedIdentityService::new(FederatedIdentityRegistry::new(adapter)));
let mapper = CoreFederatedAuthorizationMapper::new(adapter.authorization_rules());
let service = Arc::new(adapter.into_service(mapper));
(service, oidc_config_query)
}
@@ -1209,11 +1182,7 @@ mod tests {
let (resolved_oidc, resolved_oidc_query) =
resolve_federated_identity_runtime_with(Some(context.clone())).expect("context federated identity runtime");
assert!(Arc::ptr_eq(&resolved_oidc, &context_oidc));
assert!(
resolved_oidc_query
.as_ref()
.is_some_and(|query| Arc::ptr_eq(query, &context_oidc_query))
);
assert!(Arc::ptr_eq(&resolved_oidc_query, &context_oidc_query));
assert!(publish_federated_identity_runtime_with(
Some(context.clone()),
fallback_oidc.clone(),
rustfs/src/app/context/global.rs
+1 -10
View File
@@ -35,11 +35,7 @@ use super::interfaces::{
};
use crate::app::object_data_cache::ObjectDataCacheAdapter;
use crate::app::object_traffic_health::ObjectTrafficHealth;
use rustfs_iam::{
federation::{FederatedIdentityService, oidc::OidcConfigQuery},
store::object::ObjectStore,
sys::IamSys,
};
use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys};
use rustfs_kms::KmsServiceManager;
use std::sync::{Arc, OnceLock};
@@ -169,11 +165,6 @@ impl AppContext {
self.federated_identity.publish_runtime(service, oidc_config_query)
}
/// Publish a service-only federation runtime for compatibility integrations.
pub fn publish_federated_identity_service(&self, service: Arc<FederatedIdentityService>) -> bool {
self.federated_identity.publish_handle(service)
}
pub fn kms(&self) -> Arc<dyn KmsInterface> {
self.kms.clone()
}
rustfs/src/app/context/handles.rs
+14 -83
View File
@@ -32,11 +32,7 @@ use crate::config::RustFSBufferConfig;
use async_trait::async_trait;
use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials;
use rustfs_iam::{
federation::{FederatedIdentityService, oidc::OidcConfigQuery},
store::object::ObjectStore,
sys::IamSys,
};
use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
use rustfs_kms::KmsServiceManager;
use rustfs_lock::LockClient;
@@ -87,24 +83,15 @@ pub struct FederatedIdentityHandle {
struct FederatedIdentityRuntime {
service: Arc<FederatedIdentityService>,
oidc_config_query: Option<Arc<dyn OidcConfigQuery>>,
oidc_config_query: Arc<dyn OidcConfigQuery>,
}
impl FederatedIdentityHandle {
pub fn new(service: Option<Arc<FederatedIdentityService>>) -> Self {
Self {
runtime: StdRwLock::new(service.map(|service| FederatedIdentityRuntime {
service,
oidc_config_query: None,
})),
}
}
pub fn with_runtime(runtime: Option<(Arc<FederatedIdentityService>, Arc<dyn OidcConfigQuery>)>) -> Self {
Self {
runtime: StdRwLock::new(runtime.map(|(service, oidc_config_query)| FederatedIdentityRuntime {
service,
oidc_config_query: Some(oidc_config_query),
oidc_config_query,
})),
}
}
@@ -112,45 +99,26 @@ impl FederatedIdentityHandle {
impl Default for FederatedIdentityHandle {
fn default() -> Self {
Self::new(None)
Self::with_runtime(None)
}
}
impl FederatedIdentityInterface for FederatedIdentityHandle {
fn handle(&self) -> Option<Arc<FederatedIdentityService>> {
self.runtime_snapshot().map(|(service, _)| service)
}
fn oidc_config_query(&self) -> Option<Arc<dyn OidcConfigQuery>> {
self.runtime_snapshot().and_then(|(_, oidc_config_query)| oidc_config_query)
}
fn runtime_snapshot(&self) -> Option<FederatedIdentityRuntimeSnapshot> {
self.runtime.read().ok().and_then(|runtime| {
runtime
.as_ref()
.map(|runtime| (Arc::clone(&runtime.service), runtime.oidc_config_query.as_ref().map(Arc::clone)))
.map(|runtime| (Arc::clone(&runtime.service), Arc::clone(&runtime.oidc_config_query)))
})
}
fn publish_handle(&self, service: Arc<FederatedIdentityService>) -> bool {
let Ok(mut published_runtime) = self.runtime.write() else {
return false;
};
*published_runtime = Some(FederatedIdentityRuntime {
service,
oidc_config_query: None,
});
true
}
fn publish_runtime(&self, service: Arc<FederatedIdentityService>, oidc_config_query: Arc<dyn OidcConfigQuery>) -> bool {
let Ok(mut published_runtime) = self.runtime.write() else {
return false;
};
*published_runtime = Some(FederatedIdentityRuntime {
service,
oidc_config_query: Some(oidc_config_query),
oidc_config_query,
});
true
}
@@ -639,15 +607,6 @@ pub fn publish_default_federated_identity_runtime(
default_federated_identity_handle().publish_runtime(service, oidc_config_query)
}
/// Publish a service-only federation runtime for compatibility integrations.
pub fn publish_default_federated_identity_service(service: Arc<FederatedIdentityService>) -> bool {
default_federated_identity_handle().publish_handle(service)
}
pub fn federated_identity_interface(service: Option<Arc<FederatedIdentityService>>) -> Arc<dyn FederatedIdentityInterface> {
Arc::new(FederatedIdentityHandle::new(service))
}
pub fn federated_identity_runtime_interface(
runtime: Option<(Arc<FederatedIdentityService>, Arc<dyn OidcConfigQuery>)>,
) -> Arc<dyn FederatedIdentityInterface> {
@@ -686,17 +645,13 @@ pub fn default_buffer_config_interface() -> Arc<dyn BufferConfigInterface> {
mod tests {
use super::{
KmsRuntimeHandle, KmsServiceManager, ServerConfigHandle, default_federated_identity_interface,
federated_identity_interface, federated_identity_runtime_interface, publish_default_federated_identity_runtime,
runtime_sources,
federated_identity_runtime_interface, publish_default_federated_identity_runtime, runtime_sources,
};
use crate::app::context::interfaces::{KmsRuntimeInterface, ServerConfigInterface};
use rustfs_config::server_config::Config;
use rustfs_iam::{
federation::{
FederatedIdentityRegistry, FederatedIdentityService,
oidc::{OidcConfigQuery, StandardOidcAdapter},
},
oidc::OidcSys,
federation::{CoreFederatedAuthorizationMapper, FederatedIdentityService},
oidc::{OidcConfigQuery, OidcSys, StandardOidcAdapter},
};
use std::collections::HashMap;
use std::sync::Arc;
@@ -705,7 +660,8 @@ mod tests {
let oidc = OidcSys::empty().expect("empty OIDC configuration should be valid");
let adapter = Arc::new(StandardOidcAdapter::new(Arc::new(oidc)));
let oidc_config_query = adapter.clone();
let service = Arc::new(FederatedIdentityService::new(FederatedIdentityRegistry::new(adapter)));
let mapper = CoreFederatedAuthorizationMapper::new(adapter.authorization_rules());
let service = Arc::new(adapter.into_service(mapper));
(service, oidc_config_query)
}
@@ -726,7 +682,7 @@ mod tests {
assert!(Arc::ptr_eq(&resolved_query, &first_query));
let (resolved_service, resolved_query) = interface.runtime_snapshot().expect("initial federated identity runtime");
assert!(Arc::ptr_eq(&resolved_service, &first));
assert!(resolved_query.as_ref().is_some_and(|query| Arc::ptr_eq(query, &first_query)));
assert!(Arc::ptr_eq(&resolved_query, &first_query));
assert!(interface.publish_runtime(second.clone(), second_query.clone()));
assert!(
@@ -738,36 +694,11 @@ mod tests {
assert!(interface.oidc_config_query().is_some());
let (resolved_service, resolved_query) = interface.runtime_snapshot().expect("replacement federated identity runtime");
assert!(Arc::ptr_eq(&resolved_service, &second));
assert!(resolved_query.as_ref().is_some_and(|query| Arc::ptr_eq(query, &second_query)));
}
#[test]
fn federated_identity_handle_preserves_service_only_compatibility() {
let (service, _) = test_federated_identity_runtime();
let (replacement, _) = test_federated_identity_runtime();
let interface = federated_identity_interface(Some(service.clone()));
assert!(
interface
.handle()
.as_ref()
.is_some_and(|resolved| Arc::ptr_eq(resolved, &service))
);
assert!(interface.oidc_config_query().is_none());
assert!(interface.publish_handle(replacement.clone()));
assert!(
interface
.handle()
.as_ref()
.is_some_and(|resolved| Arc::ptr_eq(resolved, &replacement))
);
assert!(interface.oidc_config_query().is_none());
let (resolved_service, resolved_query) = interface.runtime_snapshot().expect("service-only federated identity runtime");
assert!(Arc::ptr_eq(&resolved_service, &replacement));
assert!(resolved_query.is_none());
assert!(Arc::ptr_eq(&resolved_query, &second_query));
}
#[test]
#[serial_test::serial(federated_identity_runtime)]
fn default_federated_identity_handle_shares_early_publish_and_replacement() {
let (first, first_query) = test_federated_identity_runtime();
let (second, second_query) = test_federated_identity_runtime();
rustfs/src/app/context/interfaces.rs
+11 -18
View File
@@ -22,11 +22,7 @@ use crate::config::RustFSBufferConfig;
use async_trait::async_trait;
use rustfs_config::server_config::Config;
use rustfs_credentials::Credentials;
use rustfs_iam::{
federation::{FederatedIdentityService, oidc::OidcConfigQuery},
store::object::ObjectStore,
sys::IamSys,
};
use rustfs_iam::{federation::FederatedIdentityService, oidc::OidcConfigQuery, store::object::ObjectStore, sys::IamSys};
use rustfs_io_metrics::{PerformanceMetrics, internode_metrics::InternodeMetrics};
use rustfs_kms::KmsServiceManager;
use rustfs_lock::LockClient;
@@ -48,25 +44,22 @@ pub trait IamInterface: Send + Sync {
}
/// Published federation runtime handles for authentication and OIDC configuration consumers.
pub type FederatedIdentityRuntimeSnapshot = (Arc<FederatedIdentityService>, Option<Arc<dyn OidcConfigQuery>>);
pub type FederatedIdentityRuntimeSnapshot = (Arc<FederatedIdentityService>, Arc<dyn OidcConfigQuery>);
pub trait FederatedIdentityInterface: Send + Sync {
fn handle(&self) -> Option<Arc<FederatedIdentityService>>;
fn runtime_snapshot(&self) -> Option<FederatedIdentityRuntimeSnapshot>;
fn handle(&self) -> Option<Arc<FederatedIdentityService>> {
self.runtime_snapshot().map(|(service, _)| service)
}
fn oidc_config_query(&self) -> Option<Arc<dyn OidcConfigQuery>> {
None
self.runtime_snapshot().map(|(_, oidc_config_query)| oidc_config_query)
}
fn runtime_snapshot(&self) -> Option<FederatedIdentityRuntimeSnapshot> {
self.handle().map(|service| (service, self.oidc_config_query()))
}
/// Compatibility publication hook for service-only implementations.
/// Standard OIDC runtimes publish the paired query through
/// [`Self::publish_runtime`].
fn publish_handle(&self, _service: Arc<FederatedIdentityService>) -> bool {
fn publish_runtime(&self, _service: Arc<FederatedIdentityService>, _oidc_config_query: Arc<dyn OidcConfigQuery>) -> bool {
false
}
fn publish_runtime(&self, service: Arc<FederatedIdentityService>, _oidc_config_query: Arc<dyn OidcConfigQuery>) -> bool {
self.publish_handle(service)
}
}
/// KMS interface for application-layer use-cases.
rustfs/src/app/lifecycle_transition_api_test.rs
+5 -1
View File
@@ -964,7 +964,11 @@ async fn copy_object_allows_new_version_for_locked_destination_but_blocks_explic
.await
.expect_err("explicit CopyObject overwrite of a locked version should be blocked");
assert_eq!(err.code(), &s3s::S3ErrorCode::AccessDenied);
assert_eq!(err.code(), &s3s::S3ErrorCode::InvalidArgument);
assert_eq!(
err.message(),
Some("Invalid argument: A destination versionId requires replication authorization")
);
assert_eq!(read_object_bytes(&ecstore, bucket.as_str(), dst_object).await, source_payload);
assert_eq!(live_object_version_count(&ecstore, bucket.as_str(), dst_object).await, 2);
}
rustfs/src/app/object/delete.rs
+9
View File
@@ -511,6 +511,15 @@ impl DefaultObjectUsecase {
// cannot flood the log.
let mut bulk_denial_logged = false;
for (idx, obj_id) in delete.objects.iter().enumerate() {
if let Err(err) = validate_object_key(&obj_id.key, "DeleteObjects") {
delete_results[idx].error = Some(s3s::dto::Error {
code: Some(err.code().as_str().to_string()),
key: Some(obj_id.key.clone()),
message: Some(err.to_string()),
version_id: obj_id.version_id.clone(),
});
continue;
}
let raw_version_id = obj_id.version_id.clone();
let (version_id, version_uuid) = match normalize_delete_objects_version_id(raw_version_id.clone()) {
Ok(parsed) => parsed,
rustfs/src/error.rs
+32
View File
@@ -695,6 +695,7 @@ impl From<StorageError> for ApiError {
StorageError::FaultyDisk
| StorageError::FaultyRemoteDisk
| StorageError::DiskNotFound
| StorageError::Timeout
| StorageError::TooManyOpenFiles => S3ErrorCode::ServiceUnavailable,
StorageError::ErasureReadQuorum | StorageError::InsufficientReadQuorum(_, _) => {
S3ErrorCode::Custom(SLOW_DOWN_READ_CODE.into())
@@ -1719,6 +1720,37 @@ mod tests {
}
}
#[test]
fn disk_quorum_timeouts_map_to_retryable_service_unavailable() {
use crate::storage_api::error::{DiskError, OBJECT_OP_IGNORED_ERRS, reduce_read_quorum_errs, reduce_write_quorum_errs};
let errors = [
Some(DiskError::Timeout),
Some(DiskError::Timeout),
Some(DiskError::Timeout),
None,
];
for reduced in [
reduce_read_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 2),
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 3),
] {
let reduced = reduced.expect("quorum of timeouts must retain its typed cause");
assert_eq!(reduced, DiskError::Timeout);
let error = ApiError::from(StorageError::from(reduced));
assert_eq!(error.code, S3ErrorCode::ServiceUnavailable);
let response = S3Error::from(error);
assert_eq!(response.status_code(), Some(http::StatusCode::SERVICE_UNAVAILABLE));
}
let degraded = [None, None, None, Some(DiskError::Timeout)];
assert!(reduce_write_quorum_errs(&degraded, OBJECT_OP_IGNORED_ERRS, 3).is_none());
for failure in [DiskError::FileCorrupt, DiskError::Io(IoError::other("unknown disk failure"))] {
let errors = [Some(failure.clone()), Some(failure.clone()), Some(failure), None];
let reduced =
reduce_write_quorum_errs(&errors, OBJECT_OP_IGNORED_ERRS, 3).expect("permanent failures must remain errors");
assert_eq!(ApiError::from(StorageError::from(reduced)).code, S3ErrorCode::InternalError);
}
}
#[test]
fn test_api_error_from_unexpected_eof_maps_to_incomplete_body() {
let io_error = IoError::new(ErrorKind::UnexpectedEof, rustfs_rio::IncompleteBody { remaining: 7 });

Some files were not shown because too many files have changed in this diff Show More